Data storage system and method based on Protobuf

Abstract

The invention discloses a data storage system and method based on Protobuf, the system comprises an I DL extension unit, a code generation unit and an operation management unit, the I DL extension unit is used for defining a data structure and introducing multi-dimensional annotation options in I DL grammar; the code generation unit is used for analyzing the I DL file with the annotation options and automatically generating data access layer codes matched with annotation semantics; and the operation management unit is used for automatically executing data storage routing, version compatibility adaptation, consistency verification, life cycle management and conflict processing according to the annotation semantics during operation. According to the method, the advantages of cross-language, light weight and high performance of the Protobuf are kept, the capability of the Protobuf in the aspects of data modeling and version management is remarkably expanded, and full-automatic closed loop from data definition to operation maintenance is achieved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data storage system and method based on Protobuf. Background Technology

[0002] In business scenarios such as game backend development, data storage has significant business characteristics: First, gameplay iterations are frequent, and the business data structure change cycle is short, typically requiring multiple updates per month. Second, version compatibility and consistency requirements are high; the version compatibility, integrity, and consistency of data are directly related to the security of players' virtual assets, and any anomaly may cause serious losses.

[0003] In traditional development models, data structure changes require developers to manually adjust code and perform extensive regression testing to ensure forward compatibility and data security. This approach is not only inefficient but also prone to errors introduced by human intervention. Furthermore, when applying data structure changes in production environments, service shutdowns are typically required to prevent external concurrent writes from compromising data consistency and integrity, severely impacting user experience.

[0004] The existing Protobuf framework, as a commonly used data serialization tool, only provides basic data type definitions and simple optional attributes at the IDL syntax level. It cannot directly express the storage semantics (such as storage media selection, sharding strategy), version compatibility strategy (such as default filling of newly added fields, handling of deprecated fields), and consistency control logic in complex business scenarios. It is difficult to adapt to the data storage needs of rapidly iterating business and cannot solve the pain points of traditional development models. Summary of the Invention

[0005] This application addresses the technical problems existing in the prior art by providing a Protobuf-based data storage system, comprising an IDL extension unit, a code generation unit, and a runtime management unit, wherein:

[0006] IDL extension units are used to define data structures and introduce multi-dimensional annotation options into the IDL syntax;

[0007] The code generation unit is used to parse the IDL file with the annotation options and automatically generate data access layer code that matches the annotation semantics;

[0008] The runtime management unit is used to automatically perform data storage routing, version compatibility adaptation, consistency verification, lifecycle management and conflict handling based on annotation semantics at runtime.

[0009] Furthermore, the annotation options include at least one of the following:

[0010] Storage type annotations are used to mark the storage medium of a field;

[0011] Lifecycle annotations are used to define the creation, update, expiration, and recycling strategies for fields;

[0012] Version compatibility strategy annotations are used to specify forward / backward compatibility rules for fields;

[0013] Security and access control annotations are used to attach field-level permission identifiers to achieve access isolation.

[0014] Furthermore, multi-dimensional annotations are introduced to be defined through external metadata files, DSLs, or database metadata tables.

[0015] Furthermore, the IDL extension unit defines one or more of the following options at the message level:

[0016] `save_type` is used to specify the split storage mode to reduce the pressure on a single table.

[0017] shard is used to enable distributed sharded storage;

[0018] The replication_factor is used to set the number of data replicas.

[0019] hot_cold_storage is used to specify the automatic tiered storage strategy for hot / cold data;

[0020] `version_policy` is used to declare a data versioning policy that supports forward / backward compatibility.

[0021] index_optimization is used to trigger the automatic index optimization mechanism.

[0022] Furthermore, in defining data structures, IDL extension units annotate fields, including:

[0023] The `@key` annotation is used to identify the primary key field and generate uniqueness verification logic.

[0024] The `field_encryption` annotation is used to independently encrypt and store sensitive fields;

[0025] The gdpr_mode annotation is used to ensure that field processing complies with privacy compliance requirements;

[0026] The `@lifecycle_pol icy` annotation is used to enable automatic expiration cleanup or archiving of fields.

[0027] Furthermore, the code generation unit includes:

[0028] Storage routing module: Based on save_type and shard configuration, automatically generate routing algorithms, shard location logic and batch write optimization;

[0029] Version management module: Combined with version_policy="forward_backward", it automatically generates bidirectional version-compatible code;

[0030] Cache and Index Optimization Module: Based on cache_strategy and index_optimization, it generates adaptive cache management and dynamic index optimization logic.

[0031] Furthermore, the operation management unit automatically performs the following operations during data writing, reading, or synchronization:

[0032] Calculate the shard location based on the primary key and trigger multi-replica synchronization;

[0033] Combining the SAVE_SPLIT strategy with a batch processing engine reduces network and disk I / O;

[0034] Based on access frequency, hot data is automatically loaded into the distributed cache, while cold data is retained in persistent storage;

[0035] The consistency check and data recovery mechanism is automatically triggered when there are version changes, node failures, or network fluctuations.

[0036] This application also provides a Protobuf-based data storage method, applied to a Protobuf-based data storage system, comprising:

[0037] IDL extension units define data structures and introduce multi-dimensional annotation options into the IDL syntax;

[0038] The code generation unit parses the IDL file with the annotation options and automatically generates data access layer code that matches the annotation semantics;

[0039] During runtime, the runtime management unit automatically performs data storage routing, version compatibility adaptation, consistency verification, lifecycle management, and conflict handling based on annotation semantics.

[0040] Furthermore, the code generation step is implemented based on a template engine, abstract syntax tree transformation, reflection mechanism, or runtime dynamic proxy.

[0041] This application also provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the Protobuf-based data storage method as described above.

[0042] In practical applications, the modules described in the systems and methods disclosed in this application can be deployed in multiple ways on a single target server, or each module can be deployed independently on different target servers. Furthermore, to provide more powerful computing capabilities, the modules can also be deployed on a cluster of target servers as needed.

[0043] Therefore, the technical solution adopted in this application achieves the following technical effects: it maintains Protobuf's advantages of cross-language compatibility, lightweight nature, and high performance, and significantly expands its capabilities in data modeling and version management, realizing a fully automated closed loop from data definition to operation and maintenance.

[0044] To provide a clearer and more comprehensive understanding of this application, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the structure of a Protobuf-based data storage system according to an embodiment of this application. Detailed Implementation

[0047] The existing Protobuf framework only provides basic data type definitions and simple optional attributes at the data description file (IDL) syntax level, which cannot directly express storage semantics, version compatibility strategies, and consistency control logic in complex business scenarios. To address this technical deficiency, this application proposes a Protobuf-based data storage system that, while maintaining Protobuf's original high performance and cross-language capabilities, expands the expressive power of the IDL, enabling it to carry richer metadata.

[0048] A Protobuf-based data storage system includes an IDL extension unit, a code generation unit, and a runtime management unit, wherein:

[0049] IDL extension units are used to define data structures and introduce multi-dimensional annotation options into the IDL syntax;

[0050] The code generation unit is used to parse the IDL file with the annotation options and automatically generate data access layer (DAL) code that matches the annotation semantics;

[0051] The runtime management unit is used to automatically perform data storage routing, version compatibility adaptation, consistency verification, lifecycle management and conflict handling based on annotation semantics at runtime.

[0052] Please see Figure 1 The technical solutions of this application will be described in detail with reference to various preferred embodiments.

[0053] This application presents a Protobuf-based data storage system, comprising an IDL extension unit, a code generation unit, and a runtime management unit, wherein:

[0054] IDL extension units are used to define data structures and introduce multi-dimensional annotation options into the IDL syntax.

[0055] The IDL extension unit introduces a multi-dimensional, composable set of annotation options based on the original Protobuf IDL syntax. These annotations include, but are not limited to:

[0056] Storage type: The tag field or the entire message is stored in memory, persistent storage, distributed cache, or a hybrid storage system.

[0057] The storage type supports five types: SAVE_MEM (memory storage), SAVE_PERSIST (persistent storage), SAVE_CACHE (distributed cache), SAVE_MIX (hybrid storage), and SAVE_SPLIT (split storage), which users can specify directly.

[0058] Lifecycle: Defines the creation, update, expiration, and recycling strategies for fields, accurate to the millisecond level or triggered by business events;

[0059] The lifecycle can be defined in the format of lifecycle(create_rule, update_rule, expire_rule, recycle_rule), where expire_rule supports millisecond-level time configuration (e.g., expire:3600000 means expiring in 1 hour) or business event configuration (e.g., expire:event=battle_end means expiring after the battle ends).

[0060] Version compatibility policy: Specifies the forward / backward compatibility rules for fields, such as the default fill policy for newly added fields and the retention and ignoring policy for obsolete fields. For example, it can be defined using the format version_policy(forward=true, backward=true, default_fill=0, obsolete_handle=retain) to explicitly set the bidirectional compatibility switch, the default value for newly added fields, and the handling method for obsolete fields.

[0061] Security and Access Control: Field-level permission identifiers can be attached to achieve access isolation for different users or modules. It can be defined using the format access_control(role=admin,encrypt=true,gdpr_mode=enable), supporting role-based permission binding, field encryption marking, and GDPR compliance mode.

[0062] In addition, the annotations also include the following:

[0063] Distributed storage annotations include shard=true / false (sharding switch), replication_factor=N (number of replicas), and hot_cold_storage=auto (automatic hot and cold storage tiering);

[0064] Performance optimization annotations include index_optimization=auto_tune (automatic index optimization), batch_write=enable (batch write), and cache_strategy=lru (caching strategy).

[0065] Table-level annotations: Add business-level comments using table_comment="xxx" for automatic generation of operations and maintenance documentation.

[0066] When defining the Protobuf message structure, users can add annotations at both the message level and the field level. For example, the following is an example of the annotation for the data structure of basic information about game players:

[0067] save_type: Used to specify the split storage mode (such as SAVE_SPLIT) to reduce the pressure on a single table;

[0068] shard: Used to enable distributed sharded storage;

[0069] replication_factor: Used to set the number of data replicas;

[0070] hot_cold_storage: Specifies the automatic tiered storage strategy for hot / cold data;

[0071] Example code is as follows:

[0072] option(save_type = SAVE_SPLIT; / / Split storage to reduce pressure on a single table.

[0073] option(shard = true; / / Sharded storage, supports horizontal scaling

[0074] option(replication_factor) = 3; / / Number of data replicas to improve fault tolerance.

[0075] option(hot_cold_storage = "auto"); / / Automatically store hot / cold data in tiers.

[0076] version_policy: Used to declare a data versioning policy that supports forward / backward compatibility (such as "forward_backward");

[0077] index_optimization: Used to trigger automatic index optimization mechanisms (such as "auto_tune").

[0078] Example code is as follows:

[0079] option(version_policy) = "forward_backward"; / / Supports bidirectional version compatibility option(index_optimization) = "auto_tune"; / / Automatic index optimization

[0080] Fine-grained annotations are available at the field level, including:

[0081] The `@key` annotation is used to identify the primary key field and generate uniqueness verification logic. Example code is as follows:

[0082] optional uint32 pid = 1 [(key) = true]; / / Player ID, primary key

[0083] optional bytes name = 2;

[0084] optional uint32 zoneid = 3;

[0085] optional uint32 sex = 4;

[0086] optional uint32 birth = 5;

[0087] optional uint32 lv = 6;

[0088] In addition, there are:

[0089] The `field_encryption` annotation is used to independently encrypt and store sensitive fields;

[0090] optional bytes name = 2[(field_encryption) = true,(gdpr_mode) = true]; / / Encryption and compliance

[0091] optional uint32 lv = 6 [(lifecycle_policy) = "expire_after_30d"]; / / Lifecycle

[0092] During the data structure design phase, this application provides users with the option to define and annotate data structures. Users can then select or combine these annotations based on their business needs. For example, in a game backend scenario, the player ID field can be marked as "globally unique, persistent storage, and strong consistency," while temporary battle data can be marked as "memory storage, short lifespan, and weak consistency."

[0093] In the example above, the player ID (pid) is marked as the primary key and has strong consistency to ensure global uniqueness and data reliability; the player name (name) is marked as encrypted storage and GDPR compliance to ensure data security and privacy; and the level (lv) is marked as expiring in 24 hours to achieve automatic cleanup of temporary data.

[0094] As can be seen from the above examples, during the message definition phase, developers can use the advanced annotation system extended by this invention to introduce multi-dimensional meta-information into the data structure, including storage strategies, sharding rules, caching mechanisms, security strategies, and lifecycle management. Among these:

[0095] save_type = SAVE_SPLIT: Specifies the split storage mode, which reduces the amount of data in a single table and optimizes I / O performance;

[0096] shard=true: Enables distributed sharded storage, combined with automatic routing algorithms, to maintain linear scalability in high-concurrency scenarios;

[0097] replication_factor=3: Multi-replica fault tolerance mechanism to ensure high availability of data security at the node level;

[0098] hot_cold_storage="auto": Intelligent hot / cold data tiered storage, automatically migrated based on access frequency;

[0099] You can also add, for example, table_comment="batch_pack", which provides business-level comments for automatically generating operation and maintenance documentation and database metadata.

[0100] Annotation information can also be defined through external metadata files (such as JSON / XML / YAML configuration), embedded domain-specific languages ​​(DSLs), or database metadata tables.

[0101] The code generation unit of this application is an improved Protobuf code generator, used to parse IDL files with the annotation options and automatically generate Data Access Layer (DAL) code that matches the annotation semantics. The generated code retains the serialization / deserialization capabilities of native Protobuf.

[0102] The code generation methods include: supporting code generation based on template engines, abstract syntax tree (AST) transformation, reflection mechanisms, or runtime dynamic proxies, with generated languages ​​covering C++, Java, Go, Python, or scripting languages.

[0103] The code generation unit includes the following modules:

[0104] Storage routing module: Based on the save_type and shard configuration, it automatically generates routing algorithms, shard location logic, and multi-replica synchronization logic based on the replication_factor annotation. Combined with the SAVE_SPLIT strategy, it optimizes batch writes and reduces I / O overhead.

[0105] Version management module: Combined with version_policy="forward_backward", it automatically generates bidirectional version compatibility code, generates automatic adaptation code when adding / deleting fields, automatically fills the default value of the new field when reading old version data, and ignores obsolete fields when writing.

[0106] Caching and index optimization module: Based on the hot_cold_storage and index_optimization annotations, it generates hot data caching logic and dynamic index adjustment code to improve data access efficiency.

[0107] The runtime management unit of this application is used to automatically perform data storage routing, version compatibility adaptation, consistency verification, lifecycle management and conflict handling, etc., based on annotation semantics at runtime.

[0108] During the operation of the business system, the generated code is used to implement full lifecycle management of data. The system automatically selects storage paths, performs data upgrades and migrations, triggers consistency checks, and performs conflict resolution and rollback operations when necessary, based on annotation information.

[0109] During business operations (data writing, reading, or synchronization), calling the generated API will automatically perform the following operations:

[0110] Calculate the shard location based on the primary key and trigger multi-replica synchronization;

[0111] Combining the SAVE_SPLIT strategy with a batch processing engine reduces network and disk I / O;

[0112] Based on access frequency, hot data is automatically loaded into the distributed cache, while cold data is retained in persistent storage;

[0113] The consistency check and data recovery mechanism is automatically triggered when there are version changes, node failures, or network fluctuations.

[0114] Consistency and integrity checks integrate data verification logic, verifying primary key uniqueness and field format validity during writing, and verifying data signature consistency during synchronization.

[0115] It also includes implementing security controls: automatically performing encrypted storage and decryption reading on fields marked with field_encryption=true, and combining the gdpr_mode annotation to ensure that data processing complies with privacy regulations.

[0116] It also includes lifecycle management: periodically cleaning up expired data based on the lifecycle_pol icy annotation, or reclaiming specified data when a business event is triggered.

[0117] It also includes exception handling: when a node fails, it automatically switches to a backup replica to provide services; when conflicts occur during data synchronization, it performs conflict resolution based on consistency level annotations (strong consistency fields use pessimistic locking, and weak consistency fields use optimistic locking), and if the conflict cannot be resolved, it triggers a rollback and an alarm.

[0118] Although this application is based on the technical solution implemented by Protobuf, its technical solution and ideas can also be applied to other serialization protocols such as FlatBuffers, Thrift, and Cap'n Proto, or self-developed binary protocols, JSON / YAML text protocols, etc.

[0119] Compared with the prior art, the present invention has the following significant advantages:

[0120] 1. Enhanced expressive power: Breaking through the limitations of native Protobuf IDL in storage semantics, during the data structure definition phase, users can annotate and mark each field or message as a whole according to actual business needs, supporting the direct description of complex storage types, lifecycles, version compatibility strategies, consistency verification rules, etc. at the data definition level.

[0121] 2. High degree of automation: The code generation process is fully automated, avoiding the costs and errors caused by manually writing compatibility processing logic.

[0122] 3. Strong compatibility and scalability: Supports forward / backward compatibility of data structures, allowing for smooth upgrades without manual migration, and is suitable for business scenarios with rapid iteration.

[0123] 4. Improved operational stability: Built-in data consistency and integrity verification mechanism effectively reduces the risk of data anomalies in a distributed environment.

[0124] 5. Wide applicability: Suitable for game backends, large-scale distributed platforms, and other systems with high applicability and strict consistency requirements.

[0125] This invention not only retains Protobuf's advantages of cross-language compatibility, lightweight design, and high performance, but also significantly expands its capabilities in data modeling and version management, achieving a fully automated closed loop from data definition to operation and maintenance.

[0126] Based on the above embodiments, this application also provides a Protobuf-based data storage method, applied to a Protobuf-based data storage system, characterized by including:

[0127] IDL extension units define data structures and introduce multi-dimensional annotation options into the IDL syntax;

[0128] The code generation unit parses the IDL file with the annotation options and automatically generates data access layer (DAL) code that matches the annotation semantics;

[0129] During runtime, the runtime management unit automatically performs data storage routing, version compatibility adaptation, consistency verification, lifecycle management, and conflict handling based on annotation semantics.

[0130] The application also provides a storage medium storing a computer program, which is executed by a processor as described in the Protobuf-based data storage system.

[0131] This application also provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to execute a Protobuf-based data storage system as described above.

[0132] It should be noted that those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, which may include, but is not limited to, read-only memory (ROM), random-access memory (RAM), magnetic disk, or optical disk.

[0133] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data storage system based on Protobuf, characterized in that, It includes an IDL extension unit, a code generation unit, and a runtime management unit, among which: IDL extension units are used to define data structures and introduce multi-dimensional annotation options into the IDL syntax; The code generation unit is used to parse the IDL file with the annotation options and automatically generate data access layer code that matches the annotation semantics; The runtime management unit is used to automatically perform data storage routing, version compatibility adaptation, consistency verification, lifecycle management and conflict handling based on annotation semantics at runtime.

2. The Protobuf-based data storage system as described in claim 1, characterized in that, The annotation options include at least one of the following: Storage type annotations are used to mark the storage medium of a field; Lifecycle annotations are used to define the creation, update, expiration, and recycling strategies for fields; Version compatibility strategy annotations are used to specify forward / backward compatibility rules for fields; Security and access control annotations are used to attach field-level permission identifiers to achieve access isolation.

3. The Protobuf-based data storage system as described in claim 1, characterized in that it introduces... Multidimensional annotations are defined through external metadata files, DSLs, or database metadata tables.

4. The Protobuf-based data storage system as described in claim 1, characterized in that, IDL extension units define one or more of the following options at the message level: `save_type` is used to specify the split storage mode to reduce the pressure on a single table. shard is used to enable distributed sharded storage; The replication_factor is used to set the number of data replicas. hot_cold_storage is used to specify the automatic tiered storage strategy for hot / cold data; `version_policy` is used to declare a data versioning policy that supports forward / backward compatibility. index_optimization is used to trigger the automatic index optimization mechanism.

5. The data storage system based on Protobuf as described in claim 1, characterized in that: In defining data structures, IDL extension units annotate fields, including: The `@key` annotation is used to identify the primary key field and generate uniqueness verification logic. The `field_encryption` annotation is used to independently encrypt and store sensitive fields; The gdpr_mode annotation is used to ensure that field processing complies with privacy compliance requirements; The `@lifecycle_pol icy` annotation is used to enable automatic expiration cleanup or archiving of fields.

6. The Protobuf-based data storage system as described in claim 1, characterized in that, The code generation unit includes: Storage routing module: Based on save_type and shard configuration, automatically generate routing algorithms, shard location logic and batch write optimization; Version management module: Combined with version_policy="forward_backward", it automatically generates bidirectional version-compatible code; Cache and Index Optimization Module: Based on cache_strategy and index_optimization, it generates adaptive cache management and dynamic index optimization logic.

7. The Protobuf-based data storage system as described in claim 1, characterized in that, The operation management unit automatically performs the following operations during data writing, reading, or synchronization: Calculate the shard location based on the primary key and trigger multi-replica synchronization; Combining the SAVE_SPLIT strategy with a batch processing engine reduces network and disk I / O; Based on access frequency, hot data is automatically loaded into the distributed cache, while cold data is retained in persistent storage; The consistency check and data recovery mechanism is automatically triggered when there are version changes, node failures, or network fluctuations.

8. A data storage method based on Protobuf, applied to a Protobuf-based data storage system, characterized in that, include: IDL extension units define data structures and introduce multi-dimensional annotation options into the IDL syntax; The code generation unit parses the IDL file with the annotation options and automatically generates data access layer code that matches the annotation semantics; During runtime, the runtime management unit automatically performs data storage routing, version compatibility adaptation, consistency verification, lifecycle management, and conflict handling based on annotation semantics.

9. The data storage method based on Protobuf as described in claim 8, characterized in that, The code generation steps are implemented based on template engines, abstract syntax tree transformations, reflection mechanisms, or runtime dynamic proxies.

10. An electronic device, characterized in that it comprises: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the Protobuf-based data storage method as described in any one of claims 8-9.

Images