Data processing method and device of lightweight blockchain

Abstract

The application discloses a data processing method and device of a lightweight blockchain, and the method comprises the following steps: dividing a data set into multiple blocks; generating a block hash and a joint hash for each block; wherein the block hash is calculated by using block data hash, and the joint hash is obtained by using the block hash and the joint hash of the previous block; wherein the joint hash of the first block is obtained by using the block hash of the first block and a set fixed string hash; encapsulating the data, the block hash and the joint hash of each block in sequence to obtain multiple blocks; storing each block in a database and establishing storage index information for each block. The application can realize the advantages of low resource occupation, low operation and maintenance threshold and high processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of blockchain technology, and in particular to a lightweight blockchain data processing method and apparatus. Background Technology

[0002] In data-sensitive business scenarios, blockchain technology is needed to ensure data immutability and traceability. Currently, mainstream blockchain platforms use complex data structures such as Merkle trees and multi-branch indexes to achieve data tamper-proofing, but these complex data structures have certain drawbacks.

[0003] For example, traditional binary Merkle trees become too deep in large-scale transaction scenarios, leading to excessively long verification paths and high computational and storage overhead. While multi-branch variants can reduce tree height, they significantly increase the hash computation of a single node and introduce caching problems. Secondly, using complex structures such as Merkle trees, multi-branch indexes, and account models requires maintaining relationships between multiple components, including block headers, transaction lists, and state databases, resulting in a complex structure. Furthermore, existing blockchain platforms require dedicated server clusters (at least 3 nodes), rely on containerized deployment tools, and require a distributed network environment, leading to high hardware costs. A single node requires ≥2GB of memory and ≥30% CPU utilization, necessitating at least 100GB of storage for block synchronization and state maintenance. Maintenance requires professional blockchain engineers and involves complex operations such as node consensus configuration, block synchronization monitoring, and network fault troubleshooting, resulting in high daily operation and maintenance costs.

[0004] Overall, existing blockchain technology solutions suffer from technical drawbacks such as high hardware costs, high resource consumption, high computing power requirements, and high operation and maintenance costs. Summary of the Invention

[0005] This invention provides a lightweight blockchain data processing method to achieve the advantages of low resource consumption, low maintenance threshold, and high processing efficiency. It can be directly adapted to lightweight deployment environments such as ordinary office PCs and small LAN servers. The method includes: Divide the dataset into multiple blocks; For each block, generate a local hash and a combined hash; the local hash is calculated using the local data hash, and the combined hash is obtained using the local hash and the combined hash of the previous block; the combined hash of the first block is obtained using the local hash of the first block and a fixed string hash. The data of each block, the local hash, and the combined hash are used sequentially to encapsulate the data, resulting in multiple blocks; Each block is stored in the database, and a storage index is created for each block.

[0006] This invention also provides a lightweight blockchain data processing device to achieve the advantages of low resource consumption, low maintenance threshold, and high processing efficiency. It can be directly adapted to lightweight deployment environments such as ordinary office PCs and small LAN servers. The device includes: The data preprocessing module is used to divide the dataset into multiple blocks; The hash calculation module is used to generate a local hash and a combined hash for each block; the local hash is obtained by calculating the local data hash, and the combined hash is obtained by combining the local hash and the combined hash of the previous block; the combined hash of the first block is obtained by combining the local hash of the first block and a fixed string hash. The encapsulation module is used to encapsulate the data of each block, the local hash, and the combined hash in sequence to obtain multiple blocks; The storage processing module is used to store each block in the database and to create storage index information for each block.

[0007] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned lightweight blockchain data processing method.

[0008] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned lightweight blockchain data processing method.

[0009] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned lightweight blockchain data processing method.

[0010] In this embodiment of the invention, the dataset is divided into multiple blocks; a local hash and a joint hash are generated for each block; the local hash is calculated using the local data hash, and the joint hash is calculated using the local hash and the joint hash of the previous block; the joint hash of the first block is obtained using the local hash of the first block and a fixed string hash; then, the data, local hash, and joint hash of each block are sequentially encapsulated to obtain multiple blocks, realizing a linear architecture of "single data block self-containment + inter-block sequential association". Each block not only encapsulates the original business data, but also has its own integrity verification information (local hash) and inter-block association information (joint hash), forming a complete data traceability link without additional components. This design allows data processing to be executed only in the order of generation, without parsing complex tree or network index relationships, reducing the computational complexity of data reading, writing, and verification from the architectural root, laying the foundation for lightweight operation. The lightweight blockchain in this embodiment reduces the complexity of the data processing structure, eliminates the need to parse the logic of multiple component associations, shortens the data processing chain, and improves processing efficiency. It is suitable for scenarios such as real-time data acquisition and high-frequency business recording. This embodiment is compatible with simple computing devices, lowering the deployment threshold to the level of general IT systems. It requires no professional cluster resources, reduces resource consumption, and lowers hardware costs by 90%. It is suitable for small office scenarios, significantly reducing the maintenance threshold and eliminating the need for specialized technical expertise, thus greatly reducing maintenance costs. Through a combination of linear architecture to reduce computational load and dual hashing to simplify the verification chain, this solution achieves lightweight advantages such as low resource consumption, low maintenance threshold, high compatibility, and high processing efficiency while ensuring the core characteristics of "data immutability and traceability." It can be directly adapted to lightweight deployment environments such as ordinary office PCs and small LAN servers, resolving the contradiction between the "heavy architecture" of traditional blockchain technology and the "lightweight requirements" of business scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart illustrating the lightweight blockchain data processing method in an embodiment of the present invention. Figure 2 This is a schematic diagram of chained hashing in an embodiment of the present invention; Figure 3 This is a schematic diagram of the file-type container structure in an embodiment of the present invention; Figure 4 This is a specific example diagram of the lightweight blockchain data processing method in an embodiment of the present invention; Figure 5 This is a schematic diagram of the data writing process in an embodiment of the present invention; Figure 6 This is a first schematic diagram of the data verification process in an embodiment of the present invention; Figure 7 This is a second schematic diagram of the data verification process in an embodiment of the present invention; Figure 8 This is a schematic diagram of the data verification process in an embodiment of the present invention; Figure 9 This is a schematic diagram of a lightweight blockchain data processing device in an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0013] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0014] In data-sensitive business scenarios, blockchain technology is needed to ensure data immutability and traceability. However, small and medium-sized business systems that rely on data processing are often deployed in local area network environments and are limited by hardware resources (such as ordinary office PCs and small servers) and operational capabilities (mostly general IT personnel without blockchain expertise), making them unable to support the complex data storage architecture of mainstream blockchain platforms. Meanwhile, the data of these small and medium-sized business systems is already stored in mature traditional databases (such as Oracle), making data migration and system reconstruction extremely costly. There is an urgent need for a lightweight data storage solution that can ensure data security and trustworthiness, seamlessly integrate with existing databases, be easy to deploy and maintain, and consume low resources, thus resolving the core contradiction between "high security requirements" and "lightweight deployment constraints."

[0015] The embodiments of this invention are based on the core principle of "simplified architecture to ensure security and compatible design to lower the threshold". Through three major technical means of "logical container encapsulation + dual hash association + lightweight verification mechanism", it maximizes the advantages of lightweight while achieving the characteristics of data immutability and traceability.

[0016] Figure 1 This is a flowchart illustrating the lightweight blockchain data processing method in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes: Step 101: Divide the dataset into multiple blocks; Step 102: Generate a local hash and a combined hash for each block; the local hash is calculated using the local data hash, and the combined hash is obtained using the local hash and the combined hash of the previous block; the combined hash of the first block is obtained using the local hash of the first block and a fixed string hash. Step 103: Sequentially encapsulate the data of each block, the local hash, and the combined hash to obtain multiple blocks; Step 104: Store each block in the database and create storage index information for each block.

[0017] The following describes in detail the data processing method of the lightweight blockchain in the embodiments of the present invention.

[0018] In practice, before dividing the dataset into multiple blocks, the method may further include: receiving raw data and temporarily storing it by category; preprocessing each category of raw data to obtain the data body (denoted as Body) as the dataset; the preprocessing includes signature verification and data cleaning; Step 101 may further include: dividing the dataset into multiple blocks according to a set storage size.

[0019] For example, receive raw data generated by actual business operations, classify and store it according to business rules, control the size of a single data block to 100KB-1MB, and include complete business fields.

[0020] Generating a block hash and a combined hash for each block can include: generating a block hash and a combined hash for each block using the SHA-256 algorithm.

[0021] The SHA-256 algorithm is used to perform a one-way hash operation on the business data (Body) of the current block to generate the block hash value (SelfCode). The block hash value (SelfCode) is used to verify the integrity of the single block data and detect whether the data has been tampered with.

[0022] The SHA-256 algorithm is used to perform a concatenated hash operation on "the current block's SelfCode + the previous block's UnionCode" to generate a union hash value (UnionCode). The union hash value (UnionCode) is used to build a chain association between data blocks to achieve integrity verification of the entire data.

[0023] In step 103, the data, local hash, and combined hash of each block are encapsulated sequentially to obtain multiple blocks. This may include: using a fixed-format structure to encapsulate the data, local hash, combined hash, and time information of each block into a block; the blocks are transmitted and stored in binary stream form.

[0024] The database structure is shown in Table 1 below. Each block is a standardized independent unit containing the following core elements. The functions and generation rules of each element are shown in Table 1 below.

[0025] Table 1

[0026] Table 1 shows the elements, data content, generation and storage rules, and core functions of each block. Referring to the element names in Table 1, a fixed-format structure is used to encapsulate the business data body, the block hash, the combined hash, the processing date, and the timestamp.

[0027] The chained hashing in this embodiment of the invention will be further explained in conjunction with the structure of each block. Figure 2 This is a schematic diagram of chain hashing in an embodiment of the present invention, referencing... Figure 2 Abandoning the traditional Merkle tree structure of blockchain, a new "sequential chain hash" is proposed to achieve data association: Block 1 (First Block): SelfCode = SHA-256(Body1), UnionCode = SHA-256(SelfCode1 + "INIT") ("INIT" is a fixed initialization string); Block2: SelfCode=SHA-256(Body2), UnionCode=SHA-256(SelfCode2+UnionCode1); Block 3: SelfCode=SHA-256(Body3), UnionCode=SHA-256(SelfCode3+UnionCode2); ... Blockn: SelfCode=SHA-256(Bodyn), UnionCode=SHA-256(SelfCoden+UnionCode(n-1)); The "anti-tampering principle" in this embodiment of the invention is as follows: If the Body2 of Block2 is modified, it will cause SelfCode2 to change, which will in turn make UnionCode2, UnionCode3, ..., UnionCoden all invalid. The tampering point can be quickly located through verification.

[0028] Compared to existing Merkle trees, chained hashing eliminates the need to maintain tree indexes and root hashes, reducing computation and storage overhead by 50%, while retaining the core feature of "tamper traceability." Furthermore, the tracing process is a linear traversal, making the logic simpler and more suitable for lightweight terminals.

[0029] In step 104, each block is stored in the database, and storage index information is created for each block.

[0030] In one embodiment, the database includes a file database and a table database; The file database is used to serialize and persistently store block headers and corresponding block bodies; The tabular database is created based on an existing relational database, with each table structure corresponding one-to-one with a block element; the block element includes block identifier, data body, local hash, combined hash, and time information.

[0031] In the lightweight blockchain system of this invention, a "container" is a storage carrier unit for blockchain data. It is an independent storage entity that encapsulates storage rules, read / write strategies, and indexing logic. Its core function is to uniformly manage the storage, verification, and access of blockchain data blocks. It mainly comes in two forms: File-based containers: using independent files as carriers, data blocks are stored sequentially in binary form and managed using a dynamic index area; Table-based containers: using database tables as carriers, data blocks are broken down into fields and stored in the table's row records, managed through table indexes.

[0032] The two container forms proposed in the storage layer of this invention, namely "file-based" and "tabular" forms, can be dynamically switched through configuration files without modifying the core code.

[0033] Input of configuration file: Use YAML or Properties format. Required field is container.type (value is FILE / TABLE); optional fields include corresponding format parameters (such as file path / database connection information) and activation timing (IMMEDIATE / RESTART / NEXT_BATCH).

[0034] Configuration file parsing output: After the configuration parsing module reads the file, it outputs the container driver loading instructions and structured runtime parameters; if there is a configuration error, it outputs an exception log.

[0035] By editing configuration items → specifying when to take effect → triggering loading (timed polling or module restart), you can switch container forms without modifying the code and adapt to different business scenarios in lightweight deployment.

[0036] Figure 3 This is a schematic diagram of the file-type container structure in an embodiment of the present invention, with reference to... Figure 3It adopts a binary file structure of "block header + data block sequence", with the file extension ".safebox". A single file supports a maximum size of 10GB. If it exceeds this size, it will be automatically split into new files (named according to ProcDate, such as 20251126.safebox). Figure 3 The file header includes a security box identifier (security box = lightweight blockchain storage container), version number and its creation time, and the total number of data blocks (i.e., the total number of blocks). The dynamic index area stores index information including the processing date (the business processing date corresponding to the block), timestamp (the precise time the block was written), address, and length. Each index item is separated by a delimiter for quick parsing and differentiation, and supports dynamic expansion to adapt to increases or decreases in the number of blocks. The index area is associated with the data block area (the main area, the core storage part), where blocks support native I / O interfaces, require no middleware dependencies, load a 1GB container in less than or equal to 1 second, and use time-series storage (arranged sequentially according to the writing order). The first 4 bytes of each block indicate its length (used for quickly locating block boundaries during reading). In terms of read / write strategies, "delayed disk flushing" (batch writing data to reduce disk writes) and memory mapping (mapping disk data to memory for direct access) are used during reads, significantly improving data read / write efficiency by optimizing disk I / O frequency.

[0037] Tabular databases directly create dedicated data tables within existing relational databases. The table structure strictly corresponds to the data block elements, requiring no custom storage engine and adapting to mainstream databases such as Oracle. Tabular containers are a form of blockchain storage layer, a blockchain data storage carrier based on a table structure. Essentially, they are a blockchain-to-database table adaptation encapsulation, belonging to the module design of the blockchain storage layer. The core is to decompose blockchain data blocks into field columns (such as processing date, hash value, business data body, etc.) and store them in database tables. The fields in the table structure are explained as follows: Body (standardized business data body, UTF-8 encoded, invalid field filtered), SelfCode (SHA-256 hash value of the Body, used for single-block integrity verification), UnionCode (cascaded hash value of the current block's SelfCode and the previous block's UnionCode, used for chain-like anti-tampering association). The table structure design is shown in Table 2 below.

[0038] Table 2

[0039] Table database adaptation and optimization processes include: Manage database connections through a database connection pool (default number of 5 connections, configurable) to avoid frequent creation / destruction of connections; Data writing adopts a hybrid mechanism of batch insertion + transaction commit, that is, each set of blocks is merged into a transaction commit. For example, when processing 10 data records, they are treated as a batch, and the batch insertion operation is performed in a single database transaction. The transaction is committed immediately after the batch is completed, which ensures data consistency and improves writing efficiency. By using a composite index of "processing date + timestamp" during queries, full table scans are avoided, improving query efficiency by 90%.

[0040] To achieve compatibility with existing databases, this embodiment of the invention employs a "data synchronization interface + field mapping" mechanism: 1. Develop a standardized data synchronization interface that supports reading business data from existing databases, and the interface is compatible with mainstream database protocols such as Oracle; 2. Configure field mapping rules to associate business fields in the existing database with the Body field of the security box data block, so as to realize the automatic extraction and encapsulation of business data; whereby the security box is a lightweight blockchain storage container, and the data block is the basic data unit in it, which is the smallest storage carrier of business data after standardization. 3. The security box's tabular container can be directly deployed in existing database instances without the need for additional dedicated storage devices, reducing the difficulty of system modification.

[0041] In this embodiment, the index information is stored in the file header of the file database or the table header of the table database. The size of the index information storage occupies less than a set percentage of the database, such as 0.1%.

[0042] In one embodiment, the method may further include, before storing each block to the database: The system includes new block verification, scheduled full verification, and abnormal data marking. New block verification includes verifying the hash of the new block and its predecessor. Scheduled full verification includes verifying the hash of all blocks.

[0043] Thus, this invention proposes three major technical means: "dual hash association + lightweight verification mechanism + logical container encapsulation," which maximize the lightweight advantage while achieving the characteristics of data immutability and traceability.

[0044] 1. Core logic: Replace "complex indexes" with "linear associations" to reduce architectural complexity.

[0045] Abandoning the multi-level indexing structures of traditional blockchains, such as Merkle trees and state databases, this architecture adopts a linear structure of "single data block self-containment + sequential association between blocks." Each data block not only encapsulates the original business data but also incorporates its own integrity verification information (block hash) and inter-block association information (joint hash), forming a complete data traceability chain without the need for additional components. This design allows data processing to be executed only in the order of generation, eliminating the need to parse complex tree or network index relationships. This reduces the computational complexity of data reading, writing, and verification from the architectural root, laying the foundation for lightweight operation.

[0046] 2. Key mechanism: Dual hash verification, balancing security and computational lightweighting.

[0047] The design employs a dual verification mechanism combining "block hash" and "joint hash" to ensure data security while avoiding redundant calculations. SelfCode: The SHA-256 algorithm is used to hash only the business data (Body) of the current data block, generating a 256-bit digest. This process only applies to a single data block, has a small computational load, and can quickly verify whether a single data block has been tampered with, avoiding repeated calculations on the entire data set.

[0048] UnionCode: Also employs the SHA-256 algorithm, but its input is a concatenated string of "current block hash + previous block union hash". This design ensures that the union hash of each data block is strongly correlated with the previous block, creating a chain structure where "a change in one block affects the whole"—if any block is tampered with, the union hash of all subsequent blocks will become invalid, achieving full data tamper-proof verification. Furthermore, the union hash calculation relies only on the single hash value of the previous block, without needing to trace earlier data, resulting in a short computational chain and low resource consumption.

[0049] 3. Compatible design: Multi-form logic container, enabling seamless integration with existing systems.

[0050] An innovative dual-format logical container, combining file and table formats, is designed. This container serves as the storage medium for data blocks, without altering the existing database architecture of the business system. File-based containers exist as a single binary file, do not depend on any database software, can be stored directly on the local disk, are suitable for simple business scenarios without an existing database, and have zero deployment threshold.

[0051] Tabular containers: Dedicated data tables can be created directly in existing relational databases. The data table fields correspond one-to-one with the five key elements of a data block: "business data, local hash, combined hash, processing date, and timestamp". The database's own indexing mechanism enables efficient querying of data blocks without the need to reconstruct the existing business data system, significantly reducing system transformation costs.

[0052] A unified container interface layer is designed to abstract and encapsulate the "read-write-index-verification" logic of file-based / table-based containers. The corresponding implementation class is dynamically loaded via a configuration file (specifying `container.type`), enabling seamless form switching without code intrusion. Regardless of the form, data blocks carry SelfCode (single-block hash) and UnionCode (chained hash). The container automatically verifies hash associations during loading to ensure data immutability.

[0053] This invention, through a combination of "linear architecture to reduce computational load, dual hashing to simplify the verification chain, and multi-form containers to reduce deployment dependencies," achieves lightweight advantages of "low resource consumption, low maintenance threshold, high compatibility, and high processing efficiency" while ensuring the core characteristics of "data immutability and traceability." It can be directly adapted to lightweight deployment environments such as ordinary office PCs and small LAN servers, thus resolving the contradiction between the heavy architecture of traditional blockchain technology and the lightweight requirements of business scenarios.

[0054] Figure 4 This is a specific example diagram of the lightweight blockchain data processing method in an embodiment of the present invention, with reference to... Figure 4 This demonstrates a lightweight blockchain storage architecture where each layer has a single responsibility and low coupling. Data flow is achieved through a simplified interface, avoiding the layered redundancy of traditional blockchains. The system receives raw data at the access layer, verifies the signature, and then proceeds to the preprocessing layer for cleaning and other processing, transforming it into a standardized data body. This data then reaches the core processing layer, where it is processed sequentially through hash calculation, data encapsulation, verification, and indexing units to obtain encapsulated data blocks (blocks), which are then stored at the storage management layer. The output and monitoring layers provide functions such as querying, monitoring, and log auditing.

[0055] Figure 4 The middle access layer, serving as the interaction point between external business systems and the blockchain, is responsible for securely and efficiently receiving external requests. The multi-protocol gateway supports multiple protocols such as HTTP, GPOC, and MQ, adapting to the communication needs of different business systems. The request rate limiting component controls the flow of high-frequency requests, preventing sudden traffic surges and ensuring service stability. The data signature verification module verifies the signature of incoming data, confirming the legality and integrity of the data source. The business system adaptation interface provides standardized integration capabilities, reducing the complexity of external business system access and supporting high-concurrency scenarios with peak QPS (queries per second) ≥ 1000.

[0056] Figure 4The preprocessing layer standardizes, cleans, and transforms the raw data, providing a unified input format for subsequent core processing. The field mapping engine maps fields from heterogeneous external business data to standard fields in the blockchain system, improving data quality and processing efficiency by filtering out null values, invalid formats, and other invalid data. The encoding conversion component uniformly converts data to UTF-8 encoding, ensuring cross-system and cross-platform compatibility.

[0057] Figure 4 The core processing layer implements the core logic of blockchain data, completing the encapsulation, verification, and hash calculation of data blocks. The hash calculation unit is used to calculate the hash of the current block, the union hash, and the hash value cache: SelfCode calculates the SHA-256 hash value of the data block body for single-block integrity verification; UnionCode calculates the concatenated hash value of the current block's SelfCode and the previous block's UnionCode, achieving chain-like tamper-proofing; the hash value cache is used to cache frequently accessed hash values, improving verification performance. The data encapsulation unit includes: generating a block header containing metadata such as version, timestamp, and hash value; writing precise timestamps to data blocks to ensure temporal consistency; and serializing data blocks into binary format to optimize storage and transmission efficiency. The verification unit performs incremental correlation verification by chaining hash correlation on newly added data blocks, and periodically performs hash verification on all data blocks, marking data blocks that fail verification as abnormal data and triggering subsequent alarms and processing. Index Unit: Used for address index management, maintaining the mapping relationship between data blocks and storage addresses, and supporting fast location; Hash Index: The index is built based on the hash value of the data block to improve data retrieval efficiency; At the same time, index snapshots are generated periodically for fault recovery and data consistency verification.

[0058] Figure 4 The middle storage management layer is responsible for the persistent storage and dynamic management of data blocks, ensuring data security and high availability. The container scheduling module schedules the creation, destruction, and switching of file-based / tabular containers, realizing the dynamic allocation of storage resources; the data sharding component performs sharded storage on large-scale data, improving read / write performance and scalability; the disaster recovery backup module regularly backs up data blocks and indexes to ensure no data loss; and the storage status monitoring layer collects real-time metrics such as storage capacity and IO performance, providing data support for the monitoring layer.

[0059] Figure 4The output and monitoring layer provides data query capabilities and end-to-end monitoring, ensuring system observability and rapid problem localization. The query gateway uniformly processes external data query requests and routes them to the corresponding storage nodes; the result formatting component converts query results into standardized formats such as JSON for easy parsing by external systems; the anomaly alarm module receives anomaly signals from each layer and triggers alarm notifications via email, SMS, etc.; the end-to-end monitoring component tracks data to locate performance bottlenecks and fault points; and the log auditing module records all operation logs for compliance auditing and problem tracing.

[0060] The method in this embodiment of the invention has no consensus node, no P2P network dependency, and can run as a single process. The access layer reuses the interfaces of existing business systems without modifying the business code; the core processing layer adopts a single-threaded asynchronous processing mechanism to avoid multi-threaded resource contention; the storage management layer supports dynamic container switching, and storage method changes can be completed without stopping the service. The overall architecture can run stably on a regular PC with 4GB of memory and a dual-core CPU.

[0061] The lightweight design is introduced from four aspects: hash operation, block encapsulation, verification, and indexing, as shown in Table 3 below.

[0062] Table 3

[0063] Table 3 shows the four core components, including their technical parameters, inputs and outputs, and lightweight design. Component interaction logic: The business adaptation layer (access layer and preprocessing layer) converts the raw data into a Body, which is then passed to the hash calculation unit to generate a double hash value. The data encapsulation unit receives the Body and the double hash value, and encapsulates it into a standard data block based on the system time. The dual verification service performs incremental verification on the new data block; if successful, the index management component records its storage address, and finally, it is written to the storage layer container. The entire data flow has no redundant steps, and the processing time for a single transaction is ≤50ms.

[0064] The data processing flow in this embodiment of the invention includes core processes such as "data writing process", "data verification process" and "data query process", and each process is designed with the principle of "low resource consumption and short processing link".

[0065] Figure 5 This is a schematic diagram of the data writing process in an embodiment of the present invention. Data writing is the core step in realizing the chain-like association of the blockchain. The process only includes 6 steps and does not involve the complex operations of consensus and packaging in traditional blockchains. Figure 5 As shown, the method includes: Submit the raw data; Field mapping standardization involves converting heterogeneous raw data from different business processes into a unified standard field format, eliminating differences in field naming and format between business systems, and ensuring consistent execution of subsequent processing logic. For example, a standard field template is defined: `proc_date` - business processing date (format: YYYYMMDD), `business_id` - unique business identifier, `content` - business data content, and `source_system` - data source system identifier. Supply chain business mapping: Transaction date → `proc_date` (format converted to 202x0120), Order number → `business_id`, Order details → `content`, Source → `source_system`. Freight business data mapping: Waybill date → `proc_date` (format remains 202x0120), Waybill number → `business_id`, Waybill content → `content`, System identifier → `source_system`. Subsequent double hash calculations, data block encapsulation, and other steps can then use a single set of logic to process all business data.

[0066] Double hashing involves generating a block hash (SelfCode) and a union hash (UnionCode) for each block. Encapsulate it into a standard data block, fill in time-related fields, and generate a binary block; Incremental verification; If the verification passes, record the storage address, write it to the corresponding container, and return a write success message. If the verification fails, an error alarm will be generated and a response will be returned.

[0067] The data writing process is branchless, with a processing time of ≤50ms, supporting 100+ concurrent writes per second. A "read-ahead caching" mechanism is used during the dual-hash calculation process. When the core service layer starts, the UnionCode of the last block of the container is cached in memory, avoiding reading the entire container for each write operation, thus reducing the hash calculation time by 60%. Finally, during the database writing process, the file database uses an "append-only" mode, directly writing new data blocks to the end of the file without moving existing data, while the table database achieves efficient writing through a batch insert interface.

[0068] Figure 6 This is a first schematic diagram of the data verification process in an embodiment of the present invention. This process is the core guarantee against tampering and adopts a combination of "incremental verification + periodic full verification" to ensure data security while avoiding the resource consumption caused by full verification. Figure 6 As shown, the method includes: Trigger verification; When a new block is written, the Body / block hash SelfCode is extracted; the SelfCode of the block hash of the Body is compared by recalculating the hash; the Union hash UnionCode is compared by recalculating the associated hash; the newly added single block verification result is generated and written to the local log; The system starts at 2:00 AM, reads data blocks in batches of 100 blocks each, performs double verification on each batch (including single-block SelfCode comparison verification and joint hash comparison verification), records abnormal information (including number and tampering type), and generates a full report; in terms of lightweight design, it adopts strategies such as batch processing, single batch memory ≤10MB, and execution during off-peak hours. At the same time, it will determine and record whether there are any abnormalities. If there are any abnormalities, it will send alerts via email and SMS.

[0069] The data verification anomaly handling mechanism in this embodiment of the invention is as follows: if data tampering is detected, the system will immediately lock the abnormal data block (marked as "unreadable and writable"), and automatically back up the original data of the abnormal block and the verification log for easy subsequent tracing; for cases where UnionCode is abnormal but SelfCode is normal, it can be determined that the preceding data has been tampered with, and the system will automatically locate the source block of the tampering, reducing the scope of manual investigation.

[0070] Figure 7 This is a second schematic diagram of the data verification process in an embodiment of the present invention. Through a dual mechanism of "existing data integrity verification + new data legality verification," the immutability of all data is ensured. The method includes: Verification can be triggered in two ways: periodic verification performs full verification of existing data blocks (Block1~n) at a preset period (e.g., daily); incremental verification is triggered when a new data block (Blockn+1) is written.

[0071] The existing data verification process is as follows: Read all data blocks (Block1~n) already stored in the system; recalculate the SHA-256 hash value (SelfCode) of each data block's Body and compare it with the SelfCode stored within the block; if they match, proceed to chain verification; if they do not match, determine "abnormal: single block tampering". Recalculate the concatenated hash using the SelfCode of the current block and the UnionCode (chain hash) of the previous block, and compare it with the UnionCode stored within the block; if they match, determine "existing data integrity"; if they do not match, determine "abnormal: chain tampering".

[0072] The new data verification link is as follows: Receive the new data block to be written; recalculate the SelfCode of the new block and compare it with the SelfCode stored in the block; if they match, proceed to inter-block association verification; if they do not match, determine "abnormal: transmission tampering". Recalculate the concatenated hash using the SelfCode of the new block and the UnionCode of the last existing data block, and compare it with the UnionCode of the new block; if they match, determine "new block is valid"; if they do not match, determine "abnormal: association is invalid".

[0073] Finally, the results are verified and the process is closed. If the existing data is complete and the newly added block is valid, the "append new block + update index" operation is executed, and the process ends normally; otherwise, a "failure report" is generated. At the same time, all abnormal branches (single block tampering, chained tampering, transmission tampering, illegal association) trigger tiered alerts (such as email + system logs), and the process ends abnormally.

[0074] In one embodiment, after establishing storage address index information for each block, the method may further include: Receive query requests; query requests include query conditions; Based on the query conditions, query the storage address index information to determine the target block and whether chain tracing is required; Perform chain tracing based on the target block and output the query results.

[0075] Figure 8 This is a schematic diagram of the data verification process in an embodiment of the present invention, with reference to... Figure 8 It achieves efficient querying based on "secondary index + chain traceability", which supports both fast location of single data blocks and full chain traceability.

[0076] Receive query requests, which include query conditions such as processing date (ProcDate), timestamp, or keywords. Users can directly enter a date range or a specific date to filter data. Timestamps are more precise query conditions, locating data at a specific point in time, and are generally used as a supplementary condition to ProcDate. Parse the query conditions and generate index statements; The storage address is obtained by matching the index table with the index statement; the index table stores the index information of the blocks. Read the target data block, specifically read the binary stream; Reverse-parse the data and convert it to a readable format; How do we determine whether full-chain traceability is necessary? If necessary, trace the preceding blocks in a chain to generate a traceability list; otherwise, organize the query results. Finally, the query results are returned, such as a query list; In terms of query efficiency improvement, single block query ≤ 10ms, and the fastest query for 100 ≤ 50ms.

[0077] The secondary index refers to an index table built based on ProcDate, timestamp, and keyword. The user workflow is as follows: Enter ProcDate (e.g., 202x0120), timestamp (e.g., 1737350400), and keyword (e.g., order number: ORD202x0120001) in the front-end interface or API. The system parses and triggers index matching to match the built secondary index table, quickly obtaining the storage address of the target data block and avoiding a full scan.

[0078] For scenarios where queries are performed by business keywords, the business adaptation layer will pre-establish a mapping relationship between business keywords and data blocks (Body) and store it in an index table to avoid traversing all data.

[0079] In terms of lightweight deployment, in addition to the "sequential chain hash" design, this invention also proposes resource management strategies and access control.

[0080] Resource management strategies are used to achieve low resource consumption through a triple mechanism of memory limiting, CPU scheduling, and I / O optimization, where: 1. Memory Limitation: A "fixed memory pool" design is adopted, which allocates a set size of cache for the generation operation of the hash and joint hash of this block, the block encapsulation operation, the index operation, etc., and periodically evicts unused cache data in the cache.

[0081] For example, during the initialization of the core processing layer, a 200MB memory pool is allocated, which is divided into a data cache area (150MB), a temporary computing area (30MB), and an index area (20MB). There is no dynamic memory allocation to avoid memory leaks and fragmentation. When the data cache area is full, the least used cached data is automatically evicted to ensure stable memory usage.

[0082] 2. CPU scheduling: Provides a low-priority thread + sleep mechanism. The priority of threads for generating hash and joint hash of this block, block encapsulation operation, index operation, etc. is set to "lower than normal business threads" to avoid preempting business resources. When there is no data processing, the thread enters a sleep state (sleep time 100ms, configurable), and the CPU utilization rate drops to close to 0%.

[0083] 3. IO optimization: File operations use a combination of "batch IO + cached IO" to reduce disk seek time; database operations reduce network IO through connection pooling and batch commit, reducing IO resource consumption by 70%.

[0084] Traditional blockchains rely on multi-node consensus mechanisms to ensure trust, but the consensus process is resource-intensive and has high latency. This invention, for closed local area network scenarios, adopts a "local chain verification + access control" approach to replace the consensus mechanism. The trust assurance principle is as follows: 1. Access Control: Data writing must go through the business system's authentication interface (supporting both password and key authentication methods). The core processing layer only accepts authentication-passed requests to prevent unauthorized data writing.

[0085] 2. Local chain verification: Through the dual hash verification of "SelfCode + UnionCode", it is ensured that the data cannot be tampered with after it is written. Local trust of data can be achieved without multi-node synchronization and consensus, reducing data processing latency from hundreds of milliseconds to tens of milliseconds.

[0086] 3. Log Traceability: All data operations (write, modify, delete) are recorded in detailed logs (including the operator, operation time, and data block number). The logs themselves are also stored using chained hashing to ensure traceability of operations and further strengthen trust guarantees. This principle breaks through the traditional understanding that "blockchain must rely on consensus mechanisms." In a closed and trusted local area network scenario, the combination of "permissions + verification + logs" ensures data trust while completely eliminating the resource consumption caused by consensus mechanisms, representing a core breakthrough in lightweight implementation. In the embodiment, the operation logs generated by the system running this solution (such as data writing, verification, querying, alarms, etc.) are automatically divided into log blocks according to a log entry threshold (or hourly). Each log block contains a batch of continuous log records, ensuring consistency between logs and business data.

[0087] In one embodiment, the method supports independent deployment on a single node, can run directly on Windows / Linux operating systems, requires no containerization tools, and is plug-and-play in a local area network environment.

[0088] This invention provides a visual operation and maintenance interface that supports one-click viewing of data verification results and automatic alarm for anomalies. General IT personnel can master it after one hour of training.

[0089] In summary, the embodiments of the present invention have the following technical effects.

[0090] 1. A new "sequential chain hash" is proposed to replace the traditional Merkle tree structure of blockchain, eliminating complex indexes, improving data read and write efficiency by more than 40%, and adapting to lightweight terminals such as ordinary PCs; 2. Supports both file-based and table-based container database formats, achieving 100% compatibility with traditional databases. System upgrades do not require rebuilding the existing data system, reducing upgrade costs by more than 75%. 3. The dual hash verification mechanism provides dual protection of "single block integrity + full chain consistency", achieving a 100% accuracy rate in data tampering detection, while reducing resource consumption during the verification process by 50%; 4. The size of data blocks is controlled according to business scenarios (100KB-1MB), and combined with the ProcDate index design for processing date, the data query response time is shortened to the millisecond level to meet the real-time requirements of business. 5. Single-node deployment requires no professional hardware or containerization tools, with a memory footprint of ≤200MB. General IT personnel can complete the operation and maintenance, and its lightweight characteristics are consistent throughout the entire process of deployment, operation, and maintenance.

[0091] This invention also provides a lightweight blockchain data processing device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to the lightweight blockchain data processing method, its implementation can refer to the implementation of the lightweight blockchain data processing method; repeated details will not be elaborated further.

[0092] Figure 9 This is a schematic diagram of a lightweight blockchain data processing device in an embodiment of the present invention, such as... Figure 9 As shown, the device 900 includes: Data preprocessing module 901 is used to divide the dataset into multiple blocks; The hash calculation module 902 is used to generate a local hash and a combined hash for each block; wherein, the local hash is obtained by calculating the local data hash, and the combined hash is obtained by combining the local hash and the combined hash of the previous block; wherein the combined hash of the first block is obtained by combining the local hash of the first block and a fixed string hash. The encapsulation module 903 is used to encapsulate the data of each block, the local hash, and the combined hash in sequence to obtain multiple blocks; The storage processing module 904 is used to store each block to the database and to create storage index information for each block.

[0093] In one embodiment, the data preprocessing module 901 is specifically used for: Before dividing the dataset into multiple blocks, receive the raw data and temporarily store it by category. Each type of raw data is preprocessed to obtain the data body as the dataset; preprocessing includes signature verification and data cleaning; Divide the dataset into multiple blocks according to the set storage size.

[0094] In one embodiment, the hash calculation module 902 is specifically used for: The SHA-256 algorithm is used to generate a block hash and a combined hash for each block.

[0095] In one embodiment, the encapsulation module 903 is specifically used for: A fixed-format structure is used to encapsulate the data, local hash, combined hash, and time information of each block into a block; the block is transmitted and stored in binary stream form.

[0096] In one embodiment, the database includes a file database and a table database; The file database is used to serialize and persistently store block headers and corresponding block bodies; The tabular database is created based on an existing relational database, with each table structure corresponding one-to-one with a block element; the block element includes block identifier, data body, local hash, combined hash, and time information.

[0097] In one embodiment, the device 900 further includes: The verification module is used to perform new block verification, timed full verification, and abnormal data marking before the storage processing module 904 stores each block to the database. Among them, new block verification includes verifying the hash of the new block and its predecessor; timed full verification includes verifying the hash of all blocks.

[0098] In one embodiment, the device 900 further includes: The query processing module is used to receive query requests after the storage processing module 904 establishes storage index information for each block. The query request includes query conditions. Based on the query conditions, the module queries the storage index information to determine the target block and whether chain tracing is required. Based on the target block, the module performs chain tracing and outputs the query results.

[0099] In one embodiment, the device supports independent deployment of a single node and plug-and-play functionality in a local area network environment.

[0100] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned lightweight blockchain data processing method.

[0101] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned lightweight blockchain data processing method.

[0102] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned lightweight blockchain data processing method.

[0103] Existing blockchain platforms employ complex data structures such as Merkle trees and multi-branch indexes, which suffer from the following drawbacks: First, they are highly complex to implement and maintain, requiring specialized technical personnel and are unsuitable for LAN deployments in small and medium-sized business systems. Second, their storage architecture has poor compatibility with traditional relational databases, making direct integration with existing business data systems extremely costly. Third, data read / write and verification processes consume significant resources, resulting in low efficiency on lightweight terminals such as ordinary PCs and failing to meet real-time business requirements. Therefore, to overcome these shortcomings, this invention proposes a novel storage architecture design that adapts to and meets the business requirements of "lightweight deployment + high security + low transformation cost." Table 4 below illustrates the technical advantages of this invention compared to existing technologies.

[0104] Table 4

[0105] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight blockchain data processing method, characterized in that, include: Divide the dataset into multiple blocks; For each block, generate a local hash and a combined hash; the local hash is calculated using the local data hash, and the combined hash is obtained using the local hash and the combined hash of the previous block; the combined hash of the first block is obtained using the local hash of the first block and a fixed string hash. The data of each block, the local hash, and the combined hash are used sequentially to encapsulate the data, resulting in multiple blocks; Each block is stored in the database, and a storage index is created for each block.

2. The method as described in claim 1, characterized in that, Before dividing the dataset into multiple blocks, the following steps are also included: Receive raw data and temporarily store it by category; Each type of raw data is preprocessed to obtain the data body as the dataset; preprocessing includes signature verification and data cleaning; The dataset is divided into multiple blocks, including: Divide the dataset into multiple blocks according to the set storage size.

3. The method as described in claim 1, characterized in that, For each block, generate a block hash and a combined hash, including: The SHA-256 algorithm is used to generate a block hash and a combined hash for each block.

4. The method as described in claim 1, characterized in that, The data of each block, its local hash, and the combined hash are sequentially used to encapsulate the data, resulting in multiple blocks, including: A fixed-format structure is used to encapsulate the data, local hash, combined hash, and time information of each block into a block; the block is transmitted and stored in binary stream form.

5. The method as described in claim 1, characterized in that, The database includes a file database and a table database; The file database is used to serialize and persistently store block headers and corresponding block bodies; The tabular database is created based on an existing relational database, with each table structure corresponding one-to-one with a block element; the block element includes block identifier, data body, local hash, combined hash, and time information.

6. The method as described in claim 1, characterized in that, Before storing each block to the database, the following steps are also included: The system includes new block verification, scheduled full verification, and abnormal data marking. New block verification includes verifying the hash of the new block and its predecessor. Scheduled full verification includes verifying the hash of all blocks.

7. The method as described in claim 1, characterized in that, After establishing storage index information for each block, the following is also included: Receive query requests; query requests include query conditions; Based on the query conditions, query the storage index information to determine the target block and whether chain tracing is required; Perform chain tracing based on the target block and output the query results.

8. The method as described in claim 1, characterized in that, The method supports independent deployment of a single node and plug-and-play functionality in a local area network environment.

9. A lightweight blockchain data processing device, characterized in that, include: The data preprocessing module is used to divide the dataset into multiple blocks; The hash calculation module is used to generate a local hash and a combined hash for each block; the local hash is obtained by calculating the local data hash, and the combined hash is obtained by combining the local hash and the combined hash of the previous block; the combined hash of the first block is obtained by combining the local hash of the first block and a fixed string hash. The encapsulation module is used to encapsulate each block sequentially using its data, local hash, and combined hash to obtain multiple blocks. The storage processing module is used to store each block in the database and create storage index information for each block.

10. The apparatus as claimed in claim 9, characterized in that, The data preprocessing module is specifically used for: Before dividing the dataset into multiple blocks, receive the raw data and temporarily store it by category. Each type of raw data is preprocessed to obtain the data body as the dataset; preprocessing includes signature verification and data cleaning; Divide the dataset into multiple blocks according to the set storage size.

11. The apparatus as claimed in claim 9, characterized in that, The hash calculation module is specifically used for: The SHA-256 algorithm is used to generate a block hash and a combined hash for each block.

12. The apparatus as claimed in claim 9, characterized in that, The encapsulation module is specifically used for: A fixed-format structure is used to encapsulate the data, local hash, combined hash, and time information of each block into a block; the block is transmitted and stored in binary stream form.

13. The apparatus as claimed in claim 9, characterized in that, The database includes a file database and a table database; The file database is used to serialize and persistently store block headers and corresponding block bodies; The tabular database is created based on an existing relational database, with each table structure corresponding one-to-one with a block element; the block element includes block identifier, data body, local hash, combined hash, and time information.

14. The apparatus as claimed in claim 9, characterized in that, Also includes: The verification module is used to perform new block verification, scheduled full verification, and abnormal data marking before the storage processing module stores each block to the database. Among them, new block verification includes verifying the hash of the new block and its predecessor; scheduled full verification includes verifying the hash of all blocks.

15. The apparatus as claimed in claim 9, characterized in that, Also includes: The query processing module is used to receive query requests after the storage processing module has established storage index information for each block; A query request includes query criteria; Based on the query conditions, query the storage index information to determine the target block and whether chain tracing is required; Perform chain tracing based on the target block and output the query results.

16. The apparatus as claimed in claim 9, characterized in that, The device supports independent deployment of a single node and plug-and-play operation in a local area network environment.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images