Log deduplication method, device, electronic device and storage medium

Abstract

The present application discloses a log deduplication method, device, electronic device and storage medium. The method includes: hashing the first hash value corresponding to the log identifier of the log into a bucket in the process for duplication detection, and obtaining a duplication detection result; the bucket is correspondingly assigned to the second hash value corresponding to the corresponding first hash value; the bucket represents a set of hash values ​​corresponding to the log identifiers of historical logs; if the duplication detection result is the first duplication detection result, the log is discarded; the first duplication detection result represents that the corresponding second hash value is repeated in the bucket.

Description

Technical Field

[0001] The present application relates to the field of computer data processing, and in particular to a log deduplication method, device, electronic device and storage medium. Background Art

[0002] Network devices, operating systems, and service programs will generate logs when they are running. The logs record relevant content such as date, time, operator, and operation behavior. In order to conduct centralized analysis of logs, logs need to be stored in a log system. In some scenarios, when a log falls into a timeout state during the process of reporting to the log system, the log will be re-reported, causing log duplication in the log system. Therefore, it is necessary to deduplicate the logs based on a cache database before the logs enter the log system. In related technologies, the use of a cache database for log deduplication has the problems of high memory cost and high delay in reporting logs to the log system. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present application is to provide a log deduplication method, device, electronic device and storage medium to solve the problems of high memory cost and high delay in reporting logs to the log system when using a cache database for log deduplication processing in related technologies.

[0004] To achieve the above purpose, the technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a log deduplication method, the method comprising:

[0006] Hashing the first hash value corresponding to the log identifier of the log into a bucket in the process for duplicate detection, and obtaining a duplicate detection result; the bucket is correspondingly assigned to the second hash value corresponding to the corresponding first hash value; wherein the bucket represents a set composed of hash values ​​corresponding to the log identifiers of the historical logs;

[0007] In the case where the duplicate processing result is the first duplicate processing result, the log is discarded; the first duplicate processing result indicates that the corresponding second hash value is repeated in the bucket.

[0008] In the above scheme, the first hash value corresponding to the log identifier of the log is hashed into a bucket in the process for duplicate detection, including:

[0009] Hash the log identifier to at least one process; each process in the at least one process is assigned a first hash value corresponding to the log identifier;

[0010] In each of the at least one process, the corresponding first hash value is hashed into a bucket for duplicate detection.

[0011] In the above solution, when the first hash value corresponding to the log identifier is hashed into a bucket in the process for duplicate detection, it includes:

[0012] Each slot in at least one slot in the bucket is traversed, and when the corresponding second hash value is determined to be repeated based on the Bloom filter corresponding to any slot, a duplicate detection result of the corresponding second hash value being repeated is obtained; wherein,

[0013] Each slot corresponds to a Bloom filter.

[0014] In the above solution, the log deduplication method also includes:

[0015] When the duplicate processing result is the second duplicate processing result, the log identifier is added to the one bucket; the second duplicate processing result indicates that the corresponding second hash value does not have duplication in the one bucket.

[0016] In the above solution, adding the log identifier to the one bucket includes:

[0017] Detecting whether a first slot exists in the bucket; the first slot indicates that a slot in the corresponding Bloom filter does not store a corresponding second hash value;

[0018] If the first slot exists in the one bucket, adding the log identifier to the Bloom filter corresponding to the first slot;

[0019] When the first slot does not exist in the one bucket, a first slot is created in the one bucket, and the log identifier is added to a Bloom filter corresponding to the first slot.

[0020] In the above solution, the log deduplication method also includes:

[0021] At every first time interval, a second slot is created in the bucket;

[0022] When the survival time of the second slot in the corresponding bucket reaches a set time, the second slot is cleared.

[0023] In the above solution, the log deduplication method also includes:

[0024] When the set trigger condition is met, all slots included in the one bucket are saved to the setting database.

[0025] The embodiment of the present application also provides a log deduplication device, the device comprising:

[0026] A duplicate detection unit, used for hashing a first hash value corresponding to a log identifier of a log into a bucket in a process for duplicate detection, and obtaining a duplicate detection result; the bucket is correspondingly assigned to a second hash value corresponding to the corresponding first hash value; wherein the bucket represents a set composed of hash values ​​corresponding to log identifiers of historical logs;

[0027] The discarding unit is used to discard the log when the duplicate detection result is the first duplicate detection result; the first duplicate detection result indicates that the corresponding second hash value is repeated in the bucket.

[0028] The embodiment of the present application further provides an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein:

[0029] The processor is used to execute the steps of any of the above methods when running the computer program.

[0030] An embodiment of the present application further provides a storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0031] In an embodiment of the present application, a duplicate detection result is obtained by hashing the first hash value corresponding to the log identifier of the log into a bucket in the process for duplicate detection; the bucket is allocated to the second hash value corresponding to the corresponding first hash value, wherein the bucket represents a set of hash values ​​corresponding to the log identifiers of historical logs; in the case where the duplicate detection result is the first duplicate detection result, the log is discarded; the first duplicate detection result represents that the corresponding second hash value is repeated in the bucket. In this way, after the log identifier is hashed once and hashed to a process, the log identifier is hashed again. After two rounds of hashing, the real duplicate detection is performed in the bucket in the process, so that the log identifier for duplicate detection is sufficiently scattered, so that the memory occupied each time the duplicate detection is performed is sufficiently small, saving the memory space required for log deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the implementation flow of the log deduplication method provided in the embodiment of the present application;

[0033] Figure 2 A schematic diagram of the implementation flow of another log deduplication method provided in an embodiment of the present application;

[0034] Figure 3 A schematic diagram of the implementation flow of another log deduplication method provided in an embodiment of the present application;

[0035] Figure 4A schematic diagram of the implementation flow of another log deduplication method provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of a log deduplication device provided in an embodiment of the present application;

[0037] Figure 6 A schematic diagram of the hardware composition structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In the related art, the deduplication processing of logs has the requirements of high throughput and low latency. The common deduplication processing method is to perform deduplication based on an external cache database such as redis. The key-value structure of the cache database can realize the log deduplication function, and can also restore the cached data when an exception occurs. However, the cache database needs to occupy a huge memory space when performing deduplication processing. Usually, the log identifier is generated by a universally unique identifier (UUID). The common UUID consists of 36 bytes. If there are 10 billion logs that need to be deduplicated, then the memory space required for the cache database to perform deduplication processing on the log identifiers corresponding to these logs is 1*1010*36Byte, which is about 335GB. In other words, the memory space required for the cache database to perform deduplication processing will increase linearly with the increase in the number of logs, and will also increase linearly with the increase in the length of the log identifier.

[0039] If a cache database is used to deduplicate each log, the latency of the cache database in deduplicating logs will increase when the number of logs is large. If a deduplication request for a single log is optimized into a batch deduplication request for logs, this optimization method cannot achieve good results when the number of logs is large due to the performance limitations of the cache database.

[0040] That is to say, in the related art, deduplication processing of logs still has the problems of high memory cost and high delay in reporting logs to the log system.

[0041] Based on this, the embodiment of the present application provides a log deduplication method, device, electronic device and storage medium, by hashing the first hash value corresponding to the log identifier of the log into a bucket in the process for duplication detection, and obtaining a duplication detection result; the bucket is allocated to the second hash value corresponding to the corresponding first hash value, wherein the bucket represents a set composed of hash values ​​corresponding to the log identifier of the historical log; in the case where the duplication detection result is the first duplication detection result, the log is discarded; the first duplication detection result represents that the corresponding second hash value is repeated in the bucket. In this way, after the log identifier is hashed once and hashed to a process, the log identifier is hashed again. After two rounds of hashing, the real duplication detection is performed in the bucket in the process, so that the log identifier for duplication detection is sufficiently scattered, so that the memory occupied each time the duplication detection is performed is sufficiently small, saving the memory space required for log deduplication.

[0042] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0043] Figure 1 The following is a schematic diagram of the implementation process of the log deduplication method provided in the embodiment of the present application. Figure 1 As shown, the method includes:

[0044] Step 101: hash the first hash value corresponding to the log identifier of the log into a bucket in the process for duplicate detection to obtain a duplicate detection result; the bucket is correspondingly assigned to the second hash value corresponding to the corresponding first hash value; wherein the bucket represents a set of hash values ​​corresponding to the log identifiers of historical logs.

[0045] In actual applications, distributed computing frameworks such as Storm, Spark, and Flink are used to process large amounts of log data. Usually, a distributed computing framework provides a streaming data access entry Source. After the large amount of log data is connected to the Source, it is calculated and processed according to the application programming interface (API) provided by the distributed computing framework.

[0046] In the embodiment of the present application, after reading the log through the Source of the distributed computing framework, the log identifier of the log is first hashed to obtain the first hash value. In the case where the Source reads multiple logs, the log identifier of each log is hashed to different execution nodes respectively, and each execution node performs duplicate detection on the corresponding log identifier; in the case where the Source reads a log, the log identifier of the log is hashed to the corresponding execution node, and the execution node performs duplicate detection on the log identifier of the log. The specific duplicate detection process is: the first hash value corresponding to the log identifier is hashed again, hashed to a bucket in the execution node, and the hash result of the first hash value in the bucket is not repeated, and then the subsequent processing is performed; if the hash result of the first hash value in the bucket is repeated, the log corresponding to the log identifier is discarded. Here, the bucket represents a set of hash values ​​corresponding to the log identifiers of historical logs. If the hash result of the corresponding first hash value is repeated in a bucket, it means that there is already a log identifier corresponding to the hash result of the first hash value in the bucket, so the log identifier of the log is repeated in the bucket, and the log needs to be discarded.

[0047] When the first hash value corresponding to the log identifier of the log does not have a duplicate in the corresponding execution node, the first hash value corresponding to the log identifier of the log is hashed into a bucket in the process for duplicate detection. The bucket is allocated to the second hash value corresponding to the corresponding first hash value.

[0048] Figure 2 A schematic diagram of the implementation flow of another log deduplication method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown:

[0049] After reading the log through the Source, the log identifier (id, identity document) of the log is hashed to the corresponding execution node (worker). An execution node includes multiple buckets (buckets), and a bucket is a linked list composed of slots (Slots). The log identifier hashed to the execution node is hashed again and hashed to the corresponding bucket in the execution node. The duplicate detection process is performed in the corresponding bucket. If the second hash result of the log identifier in the corresponding bucket is repeated, the log corresponding to the log identifier is discarded. If the second hash result of the log identifier in the corresponding bucket is not repeated, proceed to the next step of judgment.

[0050] Step 102: When the duplicate detection result is the first duplicate detection result, the log is discarded; the first duplicate detection result indicates that the corresponding second hash value is repeated in the bucket.

[0051] Here, if the obtained duplicate detection result is the first duplicate detection result, that is, the second hash value corresponding to the log identifier is repeated in the bucket, it indicates that a log identifier corresponding to the second hash value already exists in the bucket, and the log identifier of the log is repeated in the bucket. In this case, the log is discarded. Exemplarily, after the log identifier 1 is hashed, it is hashed to bucket 1 in the process. If the log identifier 1 does not exist in bucket 1, the log identifier 1 is not repeated in bucket 1. If the log identifier 1 already exists in bucket 1, the log identifier 1 is repeated in bucket 1 at this time, and the log corresponding to the log identifier 1 is discarded.

[0052] In one embodiment, the step of hashing the first hash value corresponding to the log identifier of the log into a bucket in the process for duplicate detection includes:

[0053] Hash the log identifier to at least one process; each process in the at least one process is assigned a first hash value corresponding to the log identifier;

[0054] In each of the at least one process, the corresponding first hash value is hashed into a bucket for duplicate detection.

[0055] Here, after hashing the log identifier of a log, at least one first hash value will be obtained, and each first hash value will be placed in a process respectively. In each process, the corresponding first hash value will be hashed into a bucket for duplicate detection. Exemplarily, after hashing the log identifier of a log, 5 first hash values ​​will be obtained, and these 5 first hash values ​​will be placed in 5 different processes respectively. In each of these 5 processes, the corresponding first hash value will be hashed into a bucket for duplicate detection.

[0056] By hashing the log identifier to at least one process and hashing the corresponding first hash value into a bucket in each process for duplicate detection, the hash value corresponding to the log identifier for duplicate detection can be made small enough, so that the memory occupied by the bucket in the process for duplicate detection is small enough, saving the memory required for duplicate detection.

[0057] In one embodiment, when the first hash value corresponding to the log identifier is hashed into a bucket in the process for duplicate detection, the method includes:

[0058] Each slot in at least one slot in the bucket is traversed, and when the corresponding second hash value is determined to be repeated based on the Bloom filter corresponding to any slot, a duplicate detection result of the corresponding second hash value being repeated is obtained; wherein,

[0059] Each slot corresponds to a Bloom filter.

[0060] Here, each slot corresponds to a Bloom filter. In a bucket in a process, each slot in the bucket is traversed, and the Bloom filter corresponding to each slot is subjected to duplicate detection. If the corresponding second hash value is determined to be duplicated based on the Bloom filter corresponding to any slot, then the corresponding second hash value is considered to be duplicated.

[0061] The second hash value is subjected to duplicate detection processing through the Bloom filter corresponding to each slot in the bucket. Since the second hash value is small enough, the Bloom filter for duplicate detection processing can also be small enough, thus saving the memory required for duplicate detection processing.

[0062] In one embodiment, the log deduplication method further includes:

[0063] When the duplicate processing result is the second duplicate processing result, the log identifier is added to the one bucket; the second duplicate processing result indicates that the corresponding second hash value does not have duplication in the one bucket.

[0064] Here, if the duplicate detection result obtained is the second duplicate detection result, that is, the second hash value corresponding to the log identifier does not exist in the one bucket, it means that the log identifier corresponding to the second hash value does not exist in the bucket. At this time, the log identifier is added to the one bucket.

[0065] By adding the log identifier to the bucket when the duplicate detection result is not repeated, it is possible to facilitate deduplication judgment of subsequent log identifiers based on buckets without duplication, thereby improving the accuracy of deduplication processing.

[0066] In one embodiment, adding the log identifier to the one bucket includes:

[0067] Detecting whether a first slot exists in the bucket; the first slot indicates that a slot in the corresponding Bloom filter does not store a corresponding second hash value;

[0068] If the first slot exists in the one bucket, adding the log identifier to the Bloom filter corresponding to the first slot;

[0069] When the first slot does not exist in the one bucket, a first slot is created in the one bucket, and the log identifier is added to a Bloom filter corresponding to the first slot.

[0070] Here, the slots in the bucket are generated by time. A slot is generated every certain period of time. There is no storage content in the Bloom filter corresponding to the generated slot. Such a slot is the first slot. When adding a log identifier to the bucket, first check whether there is a first slot in the bucket. If there is a first slot in the bucket, add the log identifier to the Bloom filter corresponding to the first slot. If there is no first slot in the bucket, create a first slot in the bucket and add the log identifier to the Bloom filter corresponding to the first slot.

[0071] For example, when the misjudgment probability P is 1*10 -9 , the number of log entries n is 2*10 5 When , the bitmap size of the Bloom filter The number of hash functions is In this case, the size of each slot is about 1.1 MB, and adding the log identifier of a log to the Bloom filter corresponding to the slot requires about 30 hash calculations.

[0072] The misjudgment rate P of each Bloom filter is one in a billion. By processing the log identifiers by process and two rounds of hashing, the misjudgment rate of global log deduplication is effectively reduced. The specific misjudgment rate is related to the number of processes t and the number of buckets b. The global misjudgment rate Pglobal decreases linearly with the increase of t and b. When t is 100, b is 50, and the number of slots reserved in the bucket is 1, that is, there is only one Bloom filter, Pglobal is about one in 5 trillion, which shows that the global misjudgment rate is extremely low.

[0073] By adding the log identifier to the Bloom filter corresponding to the first slot, non-duplicate log identifiers can be saved, making it easier to perform deduplication judgment on subsequent log identifiers based on non-duplicate buckets, thereby improving the accuracy of deduplication processing.

[0074] In one embodiment, the log deduplication method further includes:

[0075] At every first time interval, a second slot is created in the bucket;

[0076] When the survival time of the second slot in the corresponding bucket reaches a set time, the second slot is cleared.

[0077] Here, deduplication processing usually has a reference time, that is, deduplication processing is performed with reference to the log identifier saved by the Bloom filter corresponding to the slot in the bucket for a previous period of time. In order to ensure the accuracy of the reference object of the deduplication processing and avoid the unlimited expansion of the number of slots in the bucket, it is necessary to manage and divide the slots for deduplication processing according to time. Therefore, a second slot is created in the bucket every first time interval, and the first time interval can be one hour or two hours, which is not limited in this embodiment of the present application. When the survival time of the second slot in the corresponding bucket reaches the set time, the set time is the expiration time of the second slot, and the second slot in the bucket is cleared.

[0078] By clearing the second slot when the survival time of the second slot in the bucket reaches the set time, memory space can be saved and the accuracy of deduplication processing can be improved.

[0079] In one embodiment, the log deduplication method further includes:

[0080] When the set trigger condition is met, all slots included in the one bucket are saved to the setting database.

[0081] Here, the setting trigger condition may be exiting the distributed computing framework, and when exiting the distributed computing framework, all slots included in a bucket are saved to a setting database, and the setting database may be a cache database. Accordingly, when the distributed computing framework is started, all slots included in a bucket are saved from the setting database to the distributed computing framework.

[0082] In actual applications, the buckets and slots in the distributed computing framework are memory structures. When the distributed computing framework is restarted, the buckets and slots may be lost, so the buckets and slots need to be backed up. The specific backup method can be to save the content stored in the buckets and slots to a cache database, such as Redis. The backup can be performed at a certain time interval, or when the number of log identifiers added to the bucket reaches a set number.

[0083] By saving all the slots included in a bucket to the set database when the set trigger conditions are met, the data in the bucket can be backed up in time to prevent the loss of data in the bucket.

[0084] In the embodiment of the present application, the first hash value corresponding to the log identifier of the log is hashed into a bucket in the process for duplicate detection; the bucket is allocated to the second hash value corresponding to the corresponding first hash value, and when the duplicate detection result corresponding to the bucket indicates that the corresponding second hash value is repeated, the log is discarded. In this way, after the log identifier is hashed once and hashed to a process, the log identifier is hashed again. After two rounds of hashing, the real duplicate detection is performed in the bucket in the process, so that the log identifier for duplicate detection is sufficiently scattered, so that the memory occupied each time the duplicate detection is performed is sufficiently small, saving the memory space required for log deduplication.

[0085] Figure 3 A schematic diagram of the implementation flow of another log deduplication method provided by the application embodiment of the present application is shown in FIG. Figure 3 As shown:

[0086] The first hash value corresponding to the log identifier is hashed to a certain process, and then hashed to the corresponding bucket in the process. It is determined whether the log identifier is repeated in the corresponding bucket. If there is a duplication, the deduplication process is terminated; if there is no duplication, the log identifier is added to the corresponding bucket.

[0087] Figure 4 A schematic diagram of the implementation flow of another log deduplication method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown:

[0088] First, determine whether there are any slots in the bucket that have not been processed for duplicate detection. If not, it means that the traversal of the slots has been completed; if it still exists, take out the slot and perform duplicate detection on the slot. If the hash value of the log identifier in the slot is repeated, the deduplication process is terminated; if the hash value of the log identifier in the slot is not repeated, it means that the traversal of the slots has been completed. Calculate the corresponding slot according to the event. Here, the log is also called an event. If there is a slot in the bucket with an empty content stored in the corresponding Bloom filter, add the log identifier to the slot; if there is no such slot, create a new slot, add the slot to the bucket, and then add the log identifier to the newly created slot. When the trigger condition is met, all the slots included in the bucket are written to the remote database. If the trigger condition is not met, the deduplication process is terminated.

[0089] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a log deduplication device, Figure 5 A schematic diagram of a log deduplication device provided in an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0090] The duplicate detection unit 501 is used to hash the first hash value corresponding to the log identifier of the log into a bucket in the process for duplicate detection to obtain a duplicate detection result; the bucket corresponds to the second hash value corresponding to the corresponding first hash value; wherein the bucket represents a set composed of hash values ​​corresponding to the log identifiers of historical logs.

[0091] The discarding unit 502 is used to discard the log when the duplicate detection result is the first duplicate detection result; the first duplicate detection result indicates that the corresponding second hash value is repeated in the bucket.

[0092] In one embodiment, the duplicate detection unit 501 is also used to hash the log identifier to at least one process; each process in the at least one process is assigned a first hash value corresponding to the log identifier; in each process of the at least one process, the corresponding first hash value is hashed into a bucket for duplicate detection.

[0093] In one embodiment, the duplicate detection unit 501 is also used to traverse each slot in at least one slot in the bucket, and when it is determined based on the Bloom filter corresponding to any slot that the corresponding second hash value is repeated, a duplicate detection processing result that the corresponding second hash value is repeated is obtained; wherein each slot corresponds to a Bloom filter.

[0094] In one embodiment, the log deduplication device also includes: an adding unit, used to add the log identifier to the one bucket when the duplicate processing result is a second duplicate processing result; the second duplicate processing result indicates that the corresponding second hash value does not have duplication in the one bucket.

[0095] In one embodiment, the adding unit is also used to detect whether there is a first slot in the bucket; the first slot represents a slot in the corresponding Bloom filter that does not store the corresponding second hash value; if the first slot exists in the bucket, the log identifier is added to the Bloom filter corresponding to the first slot; if the first slot does not exist in the bucket, the first slot is created in the bucket, and the log identifier is added to the Bloom filter corresponding to the first slot.

[0096] In one embodiment, the log deduplication device further includes: a creation unit, configured to create a second slot in the bucket at a first time interval; and clear the second slot when the survival time of the second slot in the corresponding bucket reaches a set time.

[0097] In one embodiment, the log deduplication device further includes: a saving unit, configured to save all slots included in the one bucket to a setting database when a set trigger condition is met.

[0098] In actual application, the duplicate detection unit 501, the discarding unit 502, the adding unit, the creating unit, and the saving unit can be implemented by a processor in the terminal, such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU), or a programmable gate array (FPGA).

[0099] It should be noted that: when the log deduplication device provided in the above embodiment displays information, it only uses the division of the above program modules as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the log deduplication device provided in the above embodiment and the log deduplication method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0100] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the electronic equipment includes:

[0101] Communication interface 601, capable of exchanging information with other devices such as network devices;

[0102] The processor 602 is connected to the communication interface 601 to implement information exchange with other devices and is used to execute the method provided by one or more technical solutions on the terminal side when running a computer program. The computer program is stored in the memory 603.

[0103] Specifically, the processor 602 is used to hash the first hash value corresponding to the log identifier of the log into a bucket in the process for duplicate detection processing to obtain a duplicate detection processing result; the one bucket corresponds to the second hash value corresponding to the corresponding first hash value; wherein the bucket represents a set composed of hash values ​​corresponding to the log identifiers of historical logs; when the duplicate detection processing result is the first duplicate detection processing result, the log is discarded; the first duplicate detection processing result represents that the corresponding second hash value is repeated in the one bucket.

[0104] In one embodiment, the processor 602 is also used to hash the log identifier to at least one process; each process in the at least one process is assigned a first hash value corresponding to the log identifier; in each process of the at least one process, the corresponding first hash value is hashed into a bucket for duplicate detection.

[0105] In one embodiment, when the first hash value corresponding to the log identifier is hashed into a bucket in the process for duplicate detection, the processor 602 is also used to traverse each slot in at least one slot in the bucket, and when the Bloom filter corresponding to any slot determines that the corresponding second hash value is repeated, a duplicate detection result that the corresponding second hash value is repeated is obtained; wherein each slot corresponds to a Bloom filter.

[0106] In one embodiment, the processor 602 is further configured to add the log identifier to the one bucket when the duplicate processing result is a second duplicate processing result; the second duplicate processing result indicates that the corresponding second hash value does not have duplication in the one bucket.

[0107] In one embodiment, the processor 602 is also used to detect whether there is a first slot in the bucket; the first slot represents a slot in the corresponding Bloom filter that does not store the corresponding second hash value; if the first slot exists in the bucket, the log identifier is added to the Bloom filter corresponding to the first slot; if the first slot does not exist in the bucket, the first slot is created in the bucket, and the log identifier is added to the Bloom filter corresponding to the first slot.

[0108] In one embodiment, the processor 602 is further configured to create a second slot in the bucket at every first time interval; and clear the second slot when the survival time of the second slot in the corresponding bucket reaches a set time.

[0109] In one embodiment, the processor 602 is further configured to save all slots included in the one bucket to a setting database when a set trigger condition is met.

[0110] Of course, in actual application, the various components in the electronic device are coupled together through the bus system 604. It can be understood that the bus system 604 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 604 .

[0111] The memory 603 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0112] It can be understood that the memory 603 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 605 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memory.

[0113] The method disclosed in the above embodiment of the present application can be applied to the processor 602, or implemented by the processor 602. The processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 602 or an instruction in the form of software. The above processor 602 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 602 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 603, and the processor 602 reads the program in the memory 603 and completes the steps of the above method in combination with its hardware.

[0114] When the processor 602 executes the program, the corresponding processes in the various methods of the embodiments of the present application are implemented.

[0115] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 603 storing a computer program, and the computer program can be executed by a processor 602 to complete the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0116] In the several embodiments provided in the present application, it should be understood that the disclosed devices, terminals and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0117] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0118] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0119] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks or optical disks.

[0120] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0121] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A log deduplication method, characterized in that: Applied to a cache database, the method comprises: The log identifier is hashed to a process according to a first hash value corresponding to the log identifier of the log, and in the process, the log identifier is hashed to a bucket according to a second hash value corresponding to the log identifier for duplicate detection processing to obtain a duplicate detection processing result; wherein the bucket represents a set composed of log identifiers of historical logs; When the duplicate detection result is the first duplicate detection result, the log is discarded; the first duplicate detection result indicates that the log identifier is repeated in the bucket.

2. The log deduplication method according to claim 1, characterized in that: The method includes hashing the log identifier to a process according to a first hash value corresponding to the log identifier of the log, and hashing the log identifier into a bucket in the process according to a second hash value corresponding to the log identifier for duplicate detection, including: Hash the log identifier to at least one process; each process in the at least one process corresponds to a first hash value corresponding to the log identifier; In each of the at least one process, the log identifier is hashed into a bucket according to the second hash value corresponding to the log identifier to perform duplicate detection processing.

3. The log deduplication method according to claim 1 or 2, characterized in that: When the log identifier is hashed into a bucket according to the second hash value corresponding to the log identifier for duplicate detection in the process, the method includes: Each slot in at least one slot in the bucket is traversed, and when the Bloom filter corresponding to any slot determines that the second hash value corresponding to the log identifier is repeated, a duplicate detection result of the log identifier being repeated is obtained; wherein, Each slot corresponds to a Bloom filter.

4. The log deduplication method according to claim 3, characterized in that: The method further comprises: When the duplicate detection result is the second duplicate detection result, the log identifier is added to the one bucket; the second duplicate detection result indicates that the log identifier does not have duplication in the one bucket.

5. The log deduplication method according to claim 4, characterized in that: The adding the log identifier to the one bucket includes: Detecting whether there is a first slot in the bucket; the first slot represents a slot to which a log identifier is not added in the corresponding Bloom filter; If the first slot exists in the one bucket, adding the log identifier to the Bloom filter corresponding to the first slot; When the first slot does not exist in the one bucket, a first slot is created in the one bucket, and the log identifier is added to a Bloom filter corresponding to the first slot.

6. The log deduplication method according to claim 1, characterized in that: The method further comprises: At every first time interval, a second slot is created in the bucket; When the survival time of the second slot in the corresponding bucket reaches a set time, the second slot is cleared.

7. The log deduplication method according to claim 1, characterized in that: The method further comprises: When the set trigger condition is met, all slots included in the one bucket are saved to the setting database.

8. A log deduplication device, characterized in that: Set in a cache database, the device includes: A duplicate detection unit, used for hashing the log identifier to a process according to a first hash value corresponding to the log identifier of the log, and in the process, hashing the log identifier to a bucket according to a second hash value corresponding to the log identifier for duplicate detection processing to obtain a duplicate detection processing result; wherein the bucket represents a set of log identifiers of historical logs; The discarding unit is used to discard the log when the duplicate detection processing result is the first duplicate detection processing result; the first duplicate detection processing result indicates that the log identifier is repeated in the bucket.

9. An electronic device, characterized in that: include: A processor and a memory for storing a computer program that can be executed on the processor, wherein: When the processor is used to run the computer program, the steps of the method according to any one of claims 1 to 7 are performed.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images