Structured data serialization transmission method and device based on Avro

Abstract

The invention discloses a structured data serialization transmission method and device based on Avro. The method comprises the following steps: S1, acquiring a table structure and table data of a datatable to be exported from a source end database by using a data export module, dynamically constructing Schema of Avro according to the table structure, creating an Avro encoder, and serializing thetable data by using the Avro encoder to obtain a serialized result data block; S2, transmitting the serialization result data block to a data loading module through a network via the data export module; S3, deserializing the received serialized data blocks via the data loading module by using an Avro decoder to obtain original data table data, and loading the original data table data to a standbyend database.

Description

technical field [0001] The invention relates to the technical field of computer disaster recovery and backup, in particular to an Avro-based structured data serialization transmission method and device. Background technique [0002] Serialization is the process of converting the state of data objects into a format that can be stored or transmitted. The opposite of serialization is rice serialization, which converts streams into data objects. The combination of these two processes can easily store and transmit data . [0003] At present, the data export program serializes the data exported from the source database according to a certain format, and then passes the serialization result to the data loader, and the loader deserializes the received data blocks to obtain the original data and load it to the backup Database is the main process of database backup. Whether data serialization and deserialization are efficient, and whether the volume of serialized data is small enoug...

AI Technical Summary

Problems solved by technology

[0004] Traditional data serialization methods mainly include JSON and XML. XML has been abandoned in the pursuit of high performance because of its low serialization efficiency and lengthy descriptive fields. JSON uses key-value pairs to store data, and XML comparison has made great progress in data redundancy and serialization speed, and is now the mainstream data serialization method

[0005] However, JSON has many disadvantages when processing the data in the data table: 1. JSON uses key-value pairs to store data, so when the data in the data table has many rows, it will save many duplicate data column names; 2. JSON Binary data cannot be saved directly. If you need to save binary data in JSON, you usually need to base64 encode it first and then convert it into hexadecimal printable characters. This requires additional serialization time and also reduces the volume of the serialized result data. Become bigger; Third, the serialization and deserialization performance of JSON is not good enough

However, the binary message stream of protobuf is not self-describing, which makes its serialized data not easily deserialized by the loader. Using protobuf to serialize and deserialize data needs to be customized for each table in the database in advance. A proto structure description file is deployed to both the source and backup ends. Since many data tables are usually backed up when the database is backed up, and the structure of each data table is generally not predictable in advance, this makes it very troublesome to deploy proto files in advance. Using protobuf is subject to very restrictive

Images