Structured data storage method and device

Abstract

The invention discloses a structured data storage method and a structured data storage device. The structured data storage method comprises the following steps: generating identification information of a structural body to be stored; determining the type of a field to be stored in the structural body to be stored; writing the identification information of the structural body, a fixed-length data part of the structural body and a variable-length data part of the structural body in sequence to generate serialized structural body data in a storage space; and establishing unique correspondence of the identification information and a storage format of the serialized structural body. By applying the scheme provided by the invention, high efficiency close to compact binary storage can be provided by using a compact storage scheme without losing the flexibility similar to that of protobuf, so that flexible adaptation of structural body data of different versions can be realized.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a structured data storage method and device. Background technique [0002] In the fusion application of the spider architecture, LinkCache needs to store the relevant attribute data of the Link. Link data is characterized by a large number of fields, but most of the fields are short, even bit (bit) fields, so data storage efficiency has a greater impact on data performance. On the other hand, in practical applications, the schema (structure) of Link data often involves modification of fields, so it also has high requirements for data flexibility. [0003] In the prior art, the compact binary storage method has high efficiency, but its flexibility is too poor to implement dynamic modification of the schema, and cannot be applied to the actual storage application of Link data. Tag-based text serialization methods such as XML and JSON have high flexibility, but require a l...

AI Technical Summary

Problems solved by technology

[0003] In the existing technology, the efficiency of the compact binary storage method is very high, but the flexibility is too poor, and the dynamic modification of the schema cannot be achieved, and it cannot be applied to the actual storage application of Link data.

Tag-based text serialization methods such as XML and JSON have high flexibility, but require a lot of additional storage space, resulting in low storage and transmission performance; protobuf is an open source structured file provided by Google. Data packaging method, compared with XML and JSON, protobuf uses binary tags instead of text tags, which reduces the packaging capacity to a certain extent and improves storage efficiency

However, it still stores meta-information such as field name or field number, and when the content of the field itself is small, it will cause a serious waste of storage space

Images