Blockchain-based log data processing method, apparatus, device, and medium

Abstract

The application provides a log data processing method and device based on a blockchain, equipment and a medium. The method comprises: obtaining first log data; determining a target packaging granularity matched with the first log data according to the attribute of the first log data; selecting a target number of first log data with the same attribute based on the target packaging granularity; connecting and combining the target number of first log data with the same attribute to obtain target log; performing hash processing on the target log to obtain a first hash value, and uploading transaction information carrying the first hash value to a blockchain. The application can improve the utilization rate of the blockchain to a certain extent, overcome the technical problem that the massive log and the blockchain are not matched in terms of throughput, and limit the application of the blockchain.

Description

Technical Field

[0001] This application relates to the field of blockchain technology, specifically to a blockchain-based log data processing method, apparatus, device, and medium. Background Technology

[0002] With the development of blockchain technology, it can be used for log data storage in log auditing scenarios. However, blockchain technology can only store evidence for log files. Due to the throughput bottleneck of blockchain technology, when using log files for evidence storage, the massive amount of log files that need to be transmitted creates a mismatch between the throughput of the massive logs and the blockchain, limiting the application of blockchain and resulting in low utilization efficiency. Summary of the Invention

[0003] One objective of this application is to propose a blockchain-based log data processing method, apparatus, device, and medium that at least to some extent improves the utilization rate of blockchain and overcomes the technical problem of the mismatch between massive logs and blockchain in terms of throughput, which restricts the application of blockchain.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a blockchain-based log data processing method is provided, comprising:

[0006] Get the first log data;

[0007] Based on the attributes of the first log data, determine the target packaging granularity that matches the first log data;

[0008] Based on the target packaging granularity, select a target number of first log data with the same attributes;

[0009] The target number of first log data with the same attribute are concatenated and combined to obtain the target log;

[0010] The target log is hashed to obtain a first hash value, and the transaction information carrying the first hash value is uploaded to the blockchain.

[0011] According to one aspect of the embodiments of this application, a blockchain-based log data processing apparatus is provided, comprising:

[0012] The acquisition module is used to acquire the first log data;

[0013] The determination module is used to determine the target packaging granularity that matches the first log data based on the attributes of the first log data.

[0014] The selection module is used to select a target number of first log data with the same attributes based on the target packaging granularity;

[0015] The combination module is used to connect and combine the target number of first log data with the same attribute to obtain the target log;

[0016] The upload module is used to perform hash processing on the target log to obtain a first hash value, and upload the transaction information carrying the first hash value to the blockchain.

[0017] In some embodiments of this application, based on the above technical solutions, the blockchain-based log data processing device is configured as follows:

[0018] Each first log data entry in the target log is hashed to obtain a second hash value for the first log data.

[0019] The second hash value is filtered based on a preset filter to obtain the first filter data;

[0020] Transaction information is generated based on the first hash value of the target log and the first filter data;

[0021] The transaction information is uploaded to the blockchain.

[0022] In some embodiments of this application, based on the above technical solutions, the blockchain-based log data processing device is configured as follows:

[0023] The second hash value is mapped to multiple points in a bit array based on a Bloom filter;

[0024] Then, the value corresponding to the second hash value in the bit array is set to a preset value to obtain the first filter data.

[0025] In some embodiments of this application, based on the above technical solutions, the blockchain-based log data processing device is configured as follows:

[0026] If an audit request is detected, the first hash value and the first filter data are obtained from the blockchain;

[0027] The system checks whether the first hash value matches the third hash value of the log to be audited; the log to be audited is a log obtained by concatenating and combining multiple second log data that need to be audited.

[0028] If the first hash value is inconsistent with the third hash value of the log to be audited, the data that has been tampered with is determined based on the first filter data and the second log data.

[0029] If the first hash value matches the third hash value of the log to be audited, the audit ends.

[0030] In some embodiments of this application, based on the above technical solutions, the blockchain-based log data processing device is configured as follows:

[0031] Based on the preset filter, each piece of the second log data is hashed and filtered to obtain the second filter data corresponding to each piece of the second log data. The second filter data includes the fourth hash value obtained by hashing the second log data.

[0032] The fourth hash value in the second filter data is compared with the second hash value in the first filter data one by one to determine the fourth hash value that is inconsistent with the second hash value;

[0033] Based on the second log data corresponding to the inconsistent fourth hash value, the tampered data is determined.

[0034] In some embodiments of this application, based on the above technical solutions, the blockchain-based log data processing device is configured as follows:

[0035] Add a preset number of log sample data that meet the preset conditions in terms of time attributes to the preset sampling list;

[0036] Based on the level attributes of the log sample data, the target sampling ratio of the target sample data is determined;

[0037] Based on the target sampling ratio, the log sample data with later times is traversed sequentially to the log sample data with earlier times, and the log to be sampled is selected during the traversal process.

[0038] If the selected log to be sampled is not in the historical audit log database, the target block in the preset sampling list is replaced with the log to be sampled to obtain the target sampling list.

[0039] The logs to be audited are generated based on the log sample data in the target sampling list.

[0040] In some embodiments of this application, based on the above technical solutions, the blockchain-based log data processing device is configured as follows:

[0041] Determine the reference quantity corresponding to the target packaging granularity;

[0042] Detect whether the number of first log data acquired cumulatively from a preset time has reached the reference number;

[0043] If the cumulative number of first log data obtained reaches the reference number, then the reference number is taken as the target number, and the target number of first log data with the same attribute are selected.

[0044] If the cumulative number of first log data acquired does not reach the reference number, then it is detected whether the cumulative duration of acquiring the first log data from the preset time exceeds a preset threshold.

[0045] If the cumulative duration of acquiring the first log data exceeds the preset threshold, then the number of first log data acquired cumulatively from the preset time will be taken as the target number, and the target number of first log data with the same attribute will be selected.

[0046] If the cumulative duration of acquiring the first log data does not exceed the preset threshold, then the detection of whether the cumulative number of first log data acquired exceeds the reference number is repeated until the cumulative number of first log data acquired reaches the reference number, or until the cumulative duration of acquiring the first log data exceeds the preset threshold.

[0047] According to one aspect of the embodiments of this application, an electronic device is disclosed, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the methods provided in the various optional implementations described above.

[0048] According to one aspect of the embodiments of this application, a computer program medium is disclosed, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods provided in the various optional implementations described above.

[0049] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0050] In the technical solution provided in this application embodiment, based on the attributes of the first log data, a log packaging granularity matching the first log data is adopted. A target number of first log data are selected, concatenated, and combined to obtain a target log. The first hash value carrying the target log is then uploaded to the blockchain. Because the appropriate packaging granularity is matched according to the attributes of the first log data, a suitable number of first log data are selected, and their hash values ​​are calculated and uploaded to the blockchain after concatenation and combination, thus dynamically and adaptively selecting an appropriate number of first log data for uploading to the blockchain based on the attributes of the log data. Compared to directly uploading log files to the blockchain without processing, the technical solution provided in this application embodiment greatly overcomes or alleviates the technical problem of mismatch between log data upload and blockchain throughput caused by blockchain throughput bottlenecks, thereby improving the efficiency of log data upload to the blockchain.

[0051] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0052] The above and other objectives, features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0053] Figure 1 A flowchart illustrating a blockchain-based log data processing method according to Embodiment 1 of this application is shown.

[0054] Figure 2 A flowchart illustrating a blockchain-based log data processing method according to Embodiment 2 of this application is shown.

[0055] Figure 3 A schematic diagram of the detection process of the tamper detection algorithm involved in Embodiment 2 of this application is shown.

[0056] Figure 4 A schematic diagram of the overall data processing procedure involved in Embodiment 2 of this application is shown.

[0057] Figure 5 A schematic diagram of multi-party interaction involved in Embodiment 2 of this application is shown.

[0058] Figure 6 A flowchart illustrating the blockchain-based log data processing method according to Embodiment 3 of this application is shown.

[0059] Figure 7 A schematic diagram of the structure of a blockchain-based log data processing device according to Embodiment 4 of this application is shown.

[0060] Figure 8 A schematic diagram of the structure of the electronic device involved in Embodiment 5 of this application is shown. Detailed Implementation

[0061] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0062] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0063] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0064] Example 1

[0065] Figure 1 The schematic diagram illustrates a flowchart of a blockchain-based log data processing method according to Embodiment 1 of this application. The method includes:

[0066] Step S101: Obtain the first log data;

[0067] Compared to directly processing and uploading log files to the blockchain, this embodiment processes the log data before uploading it. Log data can be the data corresponding to log entries, and the log file is obtained by first adding the log data to a pre-existing file. The log file size is fixed. In big data scenarios, this method is incompatible with the throughput bottleneck characteristics of blockchain, reducing the efficiency of processing and uploading log files to the blockchain. To improve the efficiency of uploading log data to the blockchain to a certain extent, this embodiment concatenates and combines log data according to attribute matching packaging granularity, performs hash processing, and then uploads the data to the blockchain, thereby enabling dynamic packaging and uploading of log data.

[0068] The first log data is the log data that needs to be stored and verified on the blockchain. The first log data can be generated by the log producer, such as a business system.

[0069] Step S102: Determine the target packaging granularity that matches the first log data based on the attributes of the first log data;

[0070] As an optional implementation, attributes include log level or priority. Log levels include, for example, TRACE, DEBUG, WARNING, INFO, ERROR, and FATAL. Furthermore, log attributes can be customized, as long as the target packet granularity corresponding to that attribute can be identified and obtained. The target packet granularity describes the amount of log data in a data packet.

[0071] As an alternative implementation, the attributes of the log data may include the time of the log data or the category of the log data.

[0072] There is a correspondence between attributes and target packaging granularity. This correspondence can be one-to-one or many-to-one. For example, the first log data with different attributes can each have a corresponding target packaging granularity, and the target packaging granularity corresponding to the first log data with different attributes, or the target packaging granularity corresponding to the first log data with similar attributes, can be the same.

[0073] As an optional implementation, a large amount of first log data from the business system is acquired, and the large amount of first log data from the business system is divided into multiple queues according to log attributes. Each queue stores first log data with the same attributes, and based on the attributes of the first log data in each queue, a preset packaging granularity corresponding to the attribute is determined as the target packaging granularity that matches the first log data.

[0074] It is understandable that a preset mapping relationship between the packaging granularity and the attributes can be set in advance, and the preset packaging granularity that matches the attributes of the first log data can be determined as the target packaging granularity that matches the first log data based on this mapping relationship.

[0075] Step S103: Based on the target packaging granularity, select a target number of first log data with the same attributes;

[0076] The target quantity is the number of first log data to be selected, determined based on the target packaging granularity.

[0077] For example, if the packaging granularity is determined based on the attributes of the first log data, and a data packet contains 100 first log data entries, then the target quantity is 100, and 100 first log data entries with that attribute are selected.

[0078] Step S104: Connect and combine the target number of first log data with the same attribute to obtain the target log;

[0079] The target log is a log obtained by concatenating and combining first log data with the same attributes from a target number of logs.

[0080] Step S105: Hash the target log to obtain a first hash value, and upload the transaction information carrying the first hash value to the blockchain.

[0081] The first hash value is the hash value of the target log. After hashing the target log, the hash value of the target log is obtained. The transaction information carrying this first hash value is then uploaded to the blockchain for on-chain evidence storage.

[0082] As an optional implementation, a(i) represents the packing granularity of the queue at level(i). For example, a(i) = 100 means that every 100 queues at level(i) are packed into a hash block.

[0083] As an optional implementation, the block notarization module stores transaction records on a blockchain and the smart contracts built upon it. After log packages are delivered to the block notarization module, the module triggers a pre-defined process in the smart contract to store the packages on the blockchain. The block notarization module fully leverages the immutable and non-repudiable characteristics of blockchain data to guarantee the reliability of the stored log data. The total hash value contained in each package serves as an endorsement of the reliability of all log entries within that package.

[0084] In this embodiment, based on the attributes of the first log data, a log packaging granularity matching the first log data is adopted. A target number of first log data are selected, concatenated, and combined to obtain the target log. The first hash value carrying the target log is then uploaded to the blockchain. Because the packaging granularity is matched according to the attributes of the first log data, an appropriate number of first log data are selected, and their hash values ​​are calculated before being uploaded to the blockchain, the appropriate number of first log data can be dynamically and adaptively selected for uploading based on the attributes of the log data. Compared to directly uploading log files without processing, the technical solution provided in this embodiment greatly overcomes or alleviates the technical problem of mismatch between log data upload and blockchain throughput caused by blockchain throughput bottlenecks, thereby improving the efficiency of log data upload to the blockchain.

[0085] Example 2

[0086] Figure 2 A flowchart illustrating a blockchain-based log data processing method according to Embodiment 2 of this application is shown. The method includes:

[0087] Step S201: Obtain the first log data;

[0088] Step S202: Determine the target packaging granularity that matches the first log data based on the attributes of the first log data;

[0089] Step S203: Based on the target packaging granularity, select a target number of first log data with the same attributes;

[0090] Step S204: Connect and combine the target number of first log data with the same attribute to obtain the target log;

[0091] Step S205: Perform hash processing on the target log to obtain a first hash value, and perform hash processing on each first log data in the target log to obtain a second hash value of the first log data;

[0092] This embodiment overcomes the throughput bottleneck of blockchain in big data environments and improves the efficiency of log on-chain evidence storage. It also makes it easier to find tampered data in uploaded data, thus improving audit accuracy. Compared to directly storing log files on the blockchain, this embodiment uses a filter to hash and filter the target log to obtain first filter data. This first filter data and a first hash value are then stored on the blockchain. During subsequent auditing, the first filter data can be used to compare individual log entries, narrowing down the range of suspicious logs and improving audit accuracy. This provides convenience for audit evidence collection, investigation, and tracing activities.

[0093] As an optional implementation, a second hash value is obtained by hashing each first log data entry using multiple hash functions based on a preset filter. Preset filters include, for example, Bloom filters or Cuckoo filters.

[0094] Step S206: Filter the second hash value based on a preset filter to obtain the first filter data;

[0095] As an optional implementation, the second hash value is filtered based on a preset filter to obtain first filter data, including: mapping the second hash value to multiple points in a bit array based on a Bloom filter; and setting the value in the bit array corresponding to the second hash value to a preset value to obtain the first filter data.

[0096] Alternatively, a cuckoo filter can be used to achieve filtration and obtain the first filter data.

[0097] Step S207: Generate transaction information based on the first hash value of the target log and the first filter data;

[0098] Step S208: Upload the transaction information to the blockchain.

[0099] The following describes the relevant technical solutions of this embodiment in conjunction with a specific implementation method.

[0100] The granularity of data packaging is determined by the attributes of the logs. After concatenating and combining the logs, a hash operation is performed to form a data packet of that granularity. Each log data and its corresponding hash value in the data packet with the determined granularity are added to a Bloom filter set. Each log data is mapped to K points in a bit array using K hash functions, and these points are set to 1. The data set containing the hash values ​​of each log data after incorporating the calculations of the Bloom filter rules is then packaged together.

[0101] Each log record within the same package needs to be processed through a Bloom filter or a Cuckoo filter. The resulting first hash value and the first filter data are then delivered together as a transaction to the block notarization module for on-chain notarization. The purpose of using a Bloom filter or Cuckoo filter to construct the first filter data is to allow for the comparison of log records one by one during subsequent audits of individual packages if incorrect hash values ​​are found, thus narrowing down the scope of potentially problematic logs.

[0102] As an optional implementation, after uploading the transaction information carrying the first hash value to the blockchain, the method further includes: if an audit request is detected, obtaining the first hash value and the first filter data from the blockchain; detecting whether the first hash value is consistent with the third hash value of the log to be audited; the log to be audited is a log obtained by concatenating and combining multiple second log data that need to be audited; if the first hash value is inconsistent with the third hash value of the log to be audited, determining the tampered data based on the first filter data and the second log data; if the first hash value is consistent with the third hash value of the log to be audited, the audit ends.

[0103] The third hash value is the hash value obtained by hashing the log to be audited. The second log data is the log data that needs to be audited. If the first hash value and the third hash value are the same, it indicates that the log to be audited has not been tampered with. If the first hash value and the third hash value are inconsistent, it indicates that the log to be audited has been tampered with. This method is used to quickly audit the log to be audited to detect whether it has been tampered with. In order to accurately locate the tampered log, when the first hash value and the third hash value are inconsistent, the first filter data and the second log data are used to determine the tampered log data. This method can improve the efficiency and accuracy of log auditing.

[0104] As an optional implementation, the log proof information delivered by the preprocessing module to the block storage module includes two types of proof information: integrity proof and existence proof. The tamper detection algorithm during the auditing process corresponds to these two types of proof information and is divided into first-level audit and second-level audit. The purpose of first-level audit is to quickly audit the integrity proof of log packets and quickly filter out log packets without problems; if incorrect, second-level audit is initiated. Second-level audit is a supplement to the first-level audit process, helping to locate the legitimacy of individual log entries.

[0105] The detection process of the tamper detection algorithm is as follows: Figure 3 As shown, the first-level audit first inputs the integrity certificate and verifies it. If it passes, a test report is generated directly; otherwise, the second-level audit is initiated, and the existence certificates corresponding to the logs are sequentially input into the Cuckoo Filter to determine whether a single log data is valid.

[0106] As an optional implementation, determining the tampered data based on the first filter data and the second log data includes: performing hash processing and filtering processing on each piece of the second log data based on the preset filter to obtain second filter data corresponding to each piece of the second log data, wherein the second filter data includes a fourth hash value obtained by hash processing the second log data; comparing the fourth hash value in the second filter data with the second hash value in the first filter data one by one to determine the fourth hash value that is inconsistent with the second hash value; and determining the tampered data based on the second log data corresponding to the inconsistent fourth hash value.

[0107] Understandably, based on the principles of Bloom filters or Cuckoo filters, when making comparisons, the second hash value and the fourth hash value can be determined based on the values ​​of multiple points in the bit array.

[0108] As an optional implementation, before detecting whether the first hash value is consistent with the third hash value of the log information to be audited, the method further includes: adding a preset number of log sample data whose time attributes meet preset conditions to a preset sampling list; determining a target sampling ratio for the target sample data based on the level attributes of the log sample data; based on the target sampling ratio, traversing from the log sample data with later times to the log sample data with earlier times in sequence, and selecting the log to be sampled during the traversal; if the selected log to be sampled is not in the historical audit log library, replacing the target block in the preset sampling list with the log to be sampled to obtain a target sampling list; and generating the log to be audited based on the log sample data in the target sampling list.

[0109] The time attribute can be the time the log was generated, or other log-related times. A preset condition is, for example, that the log's time falls within a preset time period. The preset time period can be a fixed period of time from the current time. Alternatively, the preset time period can be any other time period. The preset sampling list is a pre-defined list used to store sampled log data. The target sampling ratio is a sampling ratio that matches the level attribute. Based on the correspondence between the preset sampling ratio and the level attribute, a corresponding preset sampling ratio can be selected as the target sampling ratio. The historical audit log database is a database used to store previously audited logs.

[0110] The above solution selects log sample data using a weighted sampling algorithm based on time attributes, level attributes, and a historical audit log library. This results in log sample data with high coverage and high auditability. By using this log sample data as the second log data for auditing, high coverage and high auditability can be achieved for the logs.

[0111] In one specific implementation, a fixed number of log packets, denoted as m, are sampled in each sampling process. Two data structures need to be maintained during sampling. The first data structure is a bounded checked_queue of length l, used to store information about historically audited log packets. After each audit, the integrity certificate of the corresponding log packet is inserted at the end of the checked_queue. When the checked_queue is full, the head integrity certificate is removed, and then new integrity certificate data is inserted. The second data structure is a sampling list, target_list, used to store the sampling results of a single sampling process.

[0112] After sampling begins, the most recent m log packets are first added to target_list. Then, the process iterates backwards, and the probability of the (m+1)th log packet from the end being selected is:

[0113] R_LEVEL*(m / (m+1));

[0114] Here, R_LEVEL represents the sampling ratio corresponding to logs at level LEVEL. If selected and not in the checked_queue, it replaces any block in the target_list.

[0115] Similarly, the probability of the (m+2)th log packet from the end being selected is:

[0116] R_LEVEL*(m / (m+2));

[0117] The probability that the (m+k)th block from the end is selected is:

[0118] R_LEVEL*(m / (m+k));

[0119] The sampling inspection module is implemented based on the above method, aiming to partially rely on auditors for sampling inspection activities while maintaining automated auditing through smart contracts. This module includes proposed sampling strategies: sampling by different quantities, sampling by different security levels and retrograde sampling, sampling by different time ranges and completely random sampling, and sampling by different weights. Among these, the automated audit process, coupled with the automated contract, improves audit efficiency.

[0120] After defining the scope of the audit logs, auditors initiate an audit challenge to the storage end, requiring it to perform a hash calculation on the relevant packages locally and send the total hash value of the packages to the audit end for comparison. If there is a discrepancy in the on-chain comparison of the total hash value of the packages, it is considered that the logs in the packages have been tampered with. At this point, it is necessary to use the attached Bloom filter to narrow down the scope of suspicious logs, requiring the storage end to send the relevant log entries to the audit end for comparison.

[0121] It is worth noting that due to the "false positive" characteristic of Bloom filters, logs that are judged as 1 by the Bloom filter may not necessarily be untampered with. Here, the Bloom filter only plays an auxiliary role in investigation and evidence collection, narrowing down the scope of suspicious logs to a single entry as much as possible. After narrowing down the scope of suspicious logs, a report and related audit data are generated to facilitate subsequent evidence collection activities.

[0122] Reference Figure 4 As shown, the overall data processing process involved in the above technical solution is described. Multiple logs at different levels are processed in two ways: firstly, logs of the same level are concatenated and combined, and then hashed using a hash function to obtain a hash value; secondly, a filter is used in conjunction with multiple hash functions to calculate and process the data, resulting in two sets of data. These two sets of data are then combined into a data packet and uploaded to the blockchain. The auditor retrieves this data packet from the blockchain and uses the filter to audit the log data that needs to be audited. The log data to be audited can be obtained from the log storage provider, which in turn can obtain the log data from the log server.

[0123] Compared to methods that only hash files and cannot provide evidence for each piece of data, or identify specific data entries after data tampering, this implementation uses a filter algorithm before packaging. Once an audit detects tampering in a data packet, the filter algorithm is used again to pinpoint the specific tampered data record. This significantly improves audit accuracy by overcoming the bottleneck of large data throughput. Furthermore, the log blockchain utilizes the incremental and tamper-proof properties of blockchain to ensure the consistency and reliability of log data.

[0124] Reference Figure 5 As shown, Figure 5 This system comprises an auditor, a smart contract, a randomized data storage module, and a log data provider. The auditor sends a request to the smart contract, initiating an audit contract for the data packet. The smart contract then requests the randomized data storage module to compare data for tampering. If tampering is detected, the module sends feedback to the smart contract. Triggered by this feedback, the smart contract initiates an audit contract for the data records within the data packet, requesting the randomized data storage module to calculate the hash value of each record based on a filter. The randomized data storage module then requests the log data provider to compare the calculated hash value with the hash value of the data, returning the specific tampered data record to the auditor. This achieves automated, efficient, fine-grained, and high-granularity log auditing, while also mitigating the blockchain's throughput bottleneck.

[0125] By adopting the above approach, while ensuring the coverage of detection rate, filters are used to maximize the accuracy of the detected suspicious logs down to each individual record. Furthermore, corresponding efficient auditing processes are designed according to the architecture, and these processes are integrated with on-chain smart contracts as much as possible to achieve automation.

[0126] In this embodiment, by processing log attributes according to packaging granularity and using filters before on-chain execution, the throughput bottleneck of the blockchain can be greatly alleviated in log auditing scenarios in big data environments. On the other hand, the audit granularity is greatly reduced to a specific log data, which improves audit efficiency and accuracy and enables precise and efficient detection of tampered log data.

[0127] Example 3

[0128] Figure 6 A flowchart illustrating a blockchain-based log data processing method according to Embodiment 3 of this application is shown. The method includes:

[0129] Step S301: Obtain the first log data;

[0130] Step S302: Determine the target packaging granularity that matches the first log data based on the attributes of the first log data;

[0131] Step S303: Determine the reference quantity corresponding to the target packaging granularity;

[0132] In this embodiment, the target quantity is determined by combining the two dimensions of packaging granularity and time, and an appropriate amount of log data is selected for on-chain in a timely manner, which greatly alleviates the technical problem of the mismatch between the blockchain's throughput bottleneck and log on-chain.

[0133] The reference quantity is the quantity corresponding to the pre-set target packaging granularity.

[0134] Step S304: Detect whether the number of first log data accumulated since the preset time has reached the reference number;

[0135] The preset time can also be a fixed time. For example, a fixed time point can be preset as the preset time point, and the number of first log data acquired cumulatively can be checked from that fixed time point to see if it has reached a reference number.

[0136] Furthermore, the preset time is the start time for acquiring the first log data within each cycle. For example, if each second is considered a cycle, then during the timing process, starting from second 0 of that 1-second cycle, it checks whether the cumulative number of first log data acquired has reached a reference number. For example, it checks whether the cumulative number of first log data acquired within 1 second has reached 100.

[0137] Step S305: If the cumulative number of first log data obtained reaches the reference number, then the reference number is taken as the target number, and the target number of first log data with the same attribute are selected;

[0138] If the cumulative number of first log data acquired reaches the reference number, then since the number of log data in the current period matches the target packaging granularity, the reference number is directly used as the target number.

[0139] Step S306: If the number of first log data acquired cumulatively does not reach the reference number, then detect whether the cumulative duration of acquiring the first log data from the preset time exceeds a preset threshold.

[0140] If the cumulative number of first log data acquired does not reach the reference quantity, the system checks whether the acquisition of first log data has timed out. If it has, to ensure timely log uploading to the blockchain, the current number of first log data acquired should be used as the target quantity. If it has not timed out, the system continues to check whether the number of first log data acquired has reached the reference quantity to match the target packaging granularity, or checks whether the cumulative acquisition of first log data has timed out. This enables timely and dynamic uploading of logs to the blockchain, ensuring that log uploading is compatible with the blockchain throughput and effectively alleviating the blockchain's throughput bottleneck.

[0141] Step S307: If the cumulative duration of acquiring the first log data exceeds the preset threshold, the number of first log data acquired from the preset time is taken as the target number, and the target number of first log data with the same attribute are selected.

[0142] The preset threshold is a pre-set value used to detect whether the accumulated time has exceeded the limit.

[0143] Step S308: If the cumulative duration of acquiring the first log data does not exceed the preset threshold, then repeatedly check whether the cumulative number of first log data acquired exceeds the reference number, until the cumulative number of first log data acquired reaches the reference number, or until the cumulative duration of acquiring the first log data exceeds the preset threshold.

[0144] Step S309: Connect and combine the target number of first log data with the same attribute to obtain the target log;

[0145] Step S310: Hash the target log to obtain a first hash value, and upload the transaction information carrying the first hash value to the blockchain.

[0146] By adopting the above method, timely processing of logs can avoid the possibility of collusion between the storage party and the log generator to tamper with them, while the dynamic slimming of on-chain evidence content effectively reduces storage capacity.

[0147] In this embodiment, an appropriate number of first log data with the same attributes are selected through two dimensions: time and packaging granularity. These logs are then processed and uploaded to the blockchain, enabling timely and dynamic logging to the blockchain. This effectively alleviates the throughput bottleneck problem of the blockchain in a big data environment.

[0148] Example 4

[0149] Figure 7 A schematic diagram of the structure of a blockchain-based log data processing device according to Embodiment 4 is shown. The blockchain-based log data processing device includes:

[0150] Module 401 is used to obtain the first log data;

[0151] The determining module 402 is used to determine the target packaging granularity that matches the first log data based on the attributes of the first log data;

[0152] The selection module 403 is used to select a target number of first log data with the same attributes based on the target packaging granularity.

[0153] Combination module 404 is used to connect and combine the target number of first log data with the same attribute to obtain the target log;

[0154] The upload module 405 is used to perform hash processing on the target log to obtain a first hash value, and upload the transaction information carrying the first hash value to the blockchain.

[0155] In one exemplary embodiment of this application, the blockchain-based log data processing device is configured as follows:

[0156] Each first log data entry in the target log is hashed to obtain a second hash value for the first log data.

[0157] The second hash value is filtered based on a preset filter to obtain the first filter data;

[0158] Transaction information is generated based on the first hash value of the target log and the first filter data;

[0159] The transaction information is uploaded to the blockchain.

[0160] In one exemplary embodiment of this application, the blockchain-based log data processing device is configured as follows:

[0161] The second hash value is mapped to multiple points in a bit array based on a Bloom filter;

[0162] Then, the value corresponding to the second hash value in the bit array is set to a preset value to obtain the first filter data.

[0163] In one exemplary embodiment of this application, the blockchain-based log data processing device is configured as follows:

[0164] If an audit request is detected, the first hash value and the first filter data are obtained from the blockchain;

[0165] The system checks whether the first hash value matches the third hash value of the log to be audited; the log to be audited is a log obtained by concatenating and combining multiple second log data that need to be audited.

[0166] If the first hash value is inconsistent with the third hash value of the log to be audited, the data that has been tampered with is determined based on the first filter data and the second log data.

[0167] If the first hash value matches the third hash value of the log to be audited, the audit ends.

[0168] In one exemplary embodiment of this application, the blockchain-based log data processing device is configured as follows:

[0169] Based on the preset filter, each piece of the second log data is hashed and filtered to obtain the second filter data corresponding to each piece of the second log data. The second filter data includes the fourth hash value obtained by hashing the second log data.

[0170] The fourth hash value in the second filter data is compared with the second hash value in the first filter data one by one to determine the fourth hash value that is inconsistent with the second hash value;

[0171] Based on the second log data corresponding to the inconsistent fourth hash value, the tampered data is determined.

[0172] In one exemplary embodiment of this application, the blockchain-based log data processing device is configured as follows:

[0173] Add a preset number of log sample data that meet the preset conditions in terms of time attributes to the preset sampling list;

[0174] Based on the level attributes of the log sample data, the target sampling ratio of the target sample data is determined;

[0175] Based on the target sampling ratio, the log sample data with later times is traversed sequentially to the log sample data with earlier times, and the log to be sampled is selected during the traversal process.

[0176] If the selected log to be sampled is not in the historical audit log database, the target block in the preset sampling list is replaced with the log to be sampled to obtain the target sampling list.

[0177] The logs to be audited are generated based on the log sample data in the target sampling list.

[0178] In one exemplary embodiment of this application, the blockchain-based log data processing device is configured as follows:

[0179] Determine the reference quantity corresponding to the target packaging granularity;

[0180] Detect whether the number of first log data acquired cumulatively from a preset time has reached the reference number;

[0181] If the cumulative number of first log data obtained reaches the reference number, then the reference number is taken as the target number, and the target number of first log data with the same attribute are selected.

[0182] If the cumulative number of first log data acquired does not reach the reference number, then it is detected whether the cumulative duration of acquiring the first log data from the preset time exceeds a preset threshold.

[0183] If the cumulative duration of acquiring the first log data exceeds the preset threshold, then the number of first log data acquired cumulatively from the preset time will be taken as the target number, and the target number of first log data with the same attribute will be selected.

[0184] If the cumulative duration of acquiring the first log data does not exceed the preset threshold, then the detection of whether the cumulative number of first log data acquired exceeds the reference number is repeated until the cumulative number of first log data acquired reaches the reference number, or until the cumulative duration of acquiring the first log data exceeds the preset threshold.

[0185] Example 5

[0186] The following is for reference. Figure 8 To describe the electronic device 50 according to an embodiment of this application. Figure 8 The electronic device 30 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0187] like Figure 8 As shown, the electronic device 50 is manifested in the form of a general-purpose computing device. The components of the electronic device 50 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, and a bus 530 connecting different system components (including storage unit 520 and processing unit 510).

[0188] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the exemplary method description section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 510 can perform actions such as... Figure 1 The steps shown are as follows.

[0189] Storage unit 520 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include a read-only memory (ROM) 5203.

[0190] Storage unit 520 may also include a program / utility 5204 having a set (at least one) program module 5205, such program module 5205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0191] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0192] Electronic device 50 can also communicate with one or more external devices 600 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 50, and / or with any device that enables electronic device 50 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 540. Furthermore, electronic device 50 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 550. As shown, network adapter 550 communicates with other modules of electronic device 50 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 50, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0193] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the embodiments of this application.

[0194] In an exemplary embodiment of this application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods described in the above method embodiments.

[0195] According to one embodiment of this application, a program product for implementing the methods in the above-described method embodiments is also provided. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0196] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0197] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0198] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0199] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as JAVA and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0200] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0201] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0202] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a server or network device, etc.) to execute the method according to the embodiments of this application.

[0203] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. A blockchain-based log data processing method, characterized in that, The method includes: Get the first log data; Based on the attributes of the first log data, determine the target packaging granularity that matches the first log data; Based on the target packaging granularity, select a target number of first log data with the same attributes; The target number of first log data with the same attribute are concatenated and combined to obtain the target log; The target log is hashed to obtain a first hash value, and the transaction information carrying the first hash value is uploaded to the blockchain. Uploading transaction information carrying the first hash value to the blockchain includes: Each first log data entry in the target log is hashed to obtain a second hash value for the first log data. The second hash value is filtered based on a preset filter to obtain the first filter data; Transaction information is generated based on the first hash value of the target log and the first filter data; The transaction information is uploaded to the blockchain; The second hash value is filtered based on a preset filter to obtain first filter data, including: The second hash value is mapped to multiple points in a bit array based on a Bloom filter; Then, the value corresponding to the second hash value in the bit array is set to a preset value to obtain the first filter data.

2. The method as described in claim 1, characterized in that, After uploading the transaction information carrying the first hash value to the blockchain, the method further includes: If an audit request is detected, the first hash value and the first filter data are obtained from the blockchain; The system checks whether the first hash value matches the third hash value of the log to be audited; the log to be audited is a log obtained by concatenating and combining multiple second log data that need to be audited. If the first hash value is inconsistent with the third hash value of the log to be audited, the data that has been tampered with is determined based on the first filter data and the second log data. If the first hash value matches the third hash value of the log to be audited, the audit ends.

3. The method as described in claim 2, characterized in that, Based on the first filter data and the second log data, the tampered data is determined, including: Based on the preset filter, each piece of the second log data is hashed and filtered to obtain the second filter data corresponding to each piece of the second log data. The second filter data includes the fourth hash value obtained by hashing the second log data. The fourth hash value in the second filter data is compared with the second hash value in the first filter data one by one to determine the fourth hash value that is inconsistent with the second hash value; Based on the second log data corresponding to the inconsistent fourth hash value, the tampered data is determined.

4. The method as described in claim 2, characterized in that, Before detecting whether the first hash value matches the third hash value of the log information to be audited, the method further includes: Add a preset number of log sample data that meet the preset conditions in terms of time attributes to the preset sampling list; Based on the level attributes of the log sample data, the target sampling ratio of the target sample data is determined; Based on the target sampling ratio, the log sample data with later times is traversed sequentially to the log sample data with earlier times, and the log to be sampled is selected during the traversal process. If the selected log to be sampled is not in the historical audit log database, the target block in the preset sampling list is replaced with the log to be sampled to obtain the target sampling list. The logs to be audited are generated based on the log sample data in the target sampling list.

5. The method as described in claim 1, characterized in that, Based on the target packaging granularity, a target number of first log data items with the same attributes are selected, including: Determine the reference quantity corresponding to the target packaging granularity; Detect whether the number of first log data acquired cumulatively from a preset time has reached the reference number; If the cumulative number of first log data obtained reaches the reference number, then the reference number is taken as the target number, and the target number of first log data with the same attribute are selected. If the cumulative number of first log data acquired does not reach the reference number, then it is detected whether the cumulative duration of acquiring the first log data from the preset time exceeds a preset threshold. If the cumulative duration of acquiring the first log data exceeds the preset threshold, then the number of first log data acquired cumulatively from the preset time will be taken as the target number, and the target number of first log data with the same attribute will be selected. If the cumulative duration of acquiring the first log data does not exceed the preset threshold, then the detection of whether the cumulative number of first log data acquired exceeds the reference number is repeated until the cumulative number of first log data acquired reaches the reference number, or until the cumulative duration of acquiring the first log data exceeds the preset threshold.

6. A blockchain-based log data processing device, characterized in that, The device includes: The acquisition module is used to acquire the first log data; The determination module is used to determine the target packaging granularity that matches the first log data based on the attributes of the first log data. The selection module is used to select a target number of first log data with the same attributes based on the target packaging granularity; The combination module is used to connect and combine the target number of first log data with the same attribute to obtain the target log; The upload module is used to perform hash processing on the target log to obtain a first hash value, and upload the transaction information carrying the first hash value to the blockchain; The upload module is further configured to: Each first log data entry in the target log is hashed to obtain a second hash value for the first log data. The second hash value is filtered based on a preset filter to obtain the first filter data; Transaction information is generated based on the first hash value of the target log and the first filter data; The transaction information is uploaded to the blockchain; The second hash value is filtered based on a preset filter to obtain first filter data, including: The second hash value is mapped to multiple points in a bit array based on a Bloom filter; Then, the value corresponding to the second hash value in the bit array is set to a preset value to obtain the first filter data.

7. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the method of any one of claims 1 to 5.

Images