Mode-free data storage system supporting asynchronous change based on hybrid storage structure

Abstract

The invention particularly relates to a modeless data storage system supporting asynchronous change based on a hybrid storage structure. The modeless data storage system supporting the asynchronous change based on the hybrid storage structure comprises a storage module, a metadata management module, a data recombination module, a data cleaning module and a workload monitoring module. According to the mode-free data storage system supporting asynchronous change based on the mixed storage structure, services are provided for the outside through the mode storage system, asynchronous mode change is automatically triggered when write-in data is not matched with a current data mode, high-performance access of mass data can be met, flexible and free automatic data mode change can be achieved, and the data storage efficiency is improved. And the performance and the stability of the system are ensured.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a modeless data storage system that supports asynchronous changes based on a hybrid storage structure. Background Art

[0002] Data management systems have always had to make a trade-off between schema-based and schema-free storage modes.

[0003] Schema-based data storage requires that the structure or schema of the data be defined before the data is stored. Most SQL databases use a schema-based storage system, which requires a clear table structure and data type. Schema-based storage can better ensure data consistency and integrity, support complex transaction processing, and is more conducive to the optimizer to select a better execution path, resulting in better and more stable database performance.

[0004] Schema-less data storage does not require pre-defined data structures and can store data in multiple formats without changing the schema. Schema-less data storage is highly flexible and can quickly adapt to changes. Especially in the Internet era, the combination with programming languages ​​such as Java Script has led to the rapid development of schema-less data storage represented by MongoDB database.

[0005] With the development of IoT technology, the number of IoT devices and applications has grown rapidly, and the demand for efficient storage, management and analysis of the massive data generated by IoT devices has also increased. Due to the huge scale of IoT data, schema-free data storage cannot meet its requirements for high-performance reading and writing; but on the other hand, the traditional method of pre-defining data structures cannot meet the rapidly changing iteration requirements of IoT. In particular, the data sources of IoT applications are often complex, and applications cannot frequently change data modes to process a variety of data sources. Schema-free writing is a requirement that the IoT must meet.

[0006] In order to support both high-performance reading and writing and flexible data schema changes, many databases support online schema changes while using schema data storage. However, synchronous online schema changes will cause a sharp drop in application performance during the change process, which will have a significant impact on application performance and stability.

[0007] Therefore, data management in the era of the Internet of Things urgently requires a storage method that can provide high-performance read and write access and support flexible data model changes.

[0008] Based on the above situation, the present invention proposes a modeless data storage system that supports asynchronous changes based on a hybrid storage structure. Summary of the invention

[0009] In order to make up for the defects of the prior art, the present invention provides a simple and efficient modeless data storage system based on a hybrid storage structure and supporting asynchronous changes.

[0010] The present invention is achieved through the following technical solutions:

[0011] A modeless data storage system based on a hybrid storage structure supporting asynchronous changes, characterized by:

[0012] It includes storage module, metadata management module, data reorganization module, data cleaning module and workload monitoring module;

[0013] The storage module includes two parts: a basic storage unit based on mode storage implementation and an extended storage unit based on modeless storage implementation;

[0014] In order to facilitate efficient addition, deletion or modification of fields, the basic storage unit adopts column storage. However, any storage system that supports mode storage is also applicable to the method proposed in the present invention.

[0015] The data in the extended storage unit is expressed in BSON or other common schema-free data formats.

[0016] The basic storage unit and the extended storage unit are associated with each other through a physical address pointer.

[0017] Although the data storage system provides modeless write and read services, the metadata of the system still contains the complete data mode information of the data table.

[0018] The data model recorded by the metadata management module contains field information of data in all tables, including field name, field type and field length;

[0019] Data schema information is obtained through explicit declaration (i.e., user declaration) or implicit inference (i.e., inference through written data).

[0020] The metadata management module is used to record the status information of each field, including storage status information and migration status information;

[0021] The storage status information is used to record the storage status of each field and determine the access mode of field data; the storage status of the field is divided into basic mode, extended mode and mixed mode;

[0022] Among them, the basic mode means that the corresponding field is stored in the basic storage unit, and the field is accessed through the basic storage unit;

[0023] The extended mode means that the corresponding fields are stored in the extended storage unit, and the fields are accessed through the extended storage unit;

[0024] The mixed mode means that the corresponding fields are stored in both the basic storage unit and the extended storage unit; when accessing, the fields are first accessed in the basic storage unit. If there is no corresponding data in the basic storage unit, the fields are accessed in the extended storage unit.

[0025] The migration status information is used to record the data migration process of each field to assist data migration; the migration status includes no migration process, migration started and migration in progress.

[0026] Since modeless storage requires more complex encoding and decoding when writing and reading, data (especially data that is frequently read and written) needs to be migrated from extended storage to basic storage as soon as possible.

[0027] The data reorganization module is responsible for migrating data from the extended storage unit to the basic storage unit; during the migration process, the system can still provide uninterrupted read and write services to the outside world;

[0028] The data migration process is as follows:

[0029] Step S1: Migration starts

[0030] Update the migration status information of the corresponding field in the metadata, marking it as "migrating";

[0031] Step S2: Create the migration target field

[0032] Allocate storage space for the fields that need to be migrated in the basic storage unit; for example, in column storage, create a new data file and allocate storage space for each field that needs to be migrated.

[0033] Step S3: Data conversion

[0034] Reading a data snapshot of the data to be migrated from the extended storage unit, parsing the corresponding fields to be migrated from the data snapshot, and storing them in the migration target field file established in the basic storage unit in step S2;

[0035] If the storage status information of the corresponding field in the data mode is "mixed mode" before the migration starts, the data is read from the old field in the basic storage unit and stored in the new field;

[0036] During the process of establishing the migration target field and data conversion, data access on the to-be-migrated field is still provided by the extended storage unit (in extended mode), or by the basic storage and extended storage jointly (in hybrid mode).

[0037] If there is data written after the data snapshot time point, it is appended to the basic storage unit through WAL (Write-ahead Log).

[0038] Step S4: Migration takes effect

[0039] Update the information of the field to be migrated in the data model of the metadata, marking it as being served by the basic storage unit rather than the extended storage unit; after the migration takes effect, data access on the migrated field is provided by the basic storage unit.

[0040] When the data model is changed, there will be unused data in the extension area. The data cleaning module is responsible for updating the BSON data in the extension area and removing the migrated fields;

[0041] The data cleaning module realizes data updating by creating a new record that does not contain useless data in the extended storage unit and pointing the pointer in the basic storage unit to the new record.

[0042] The urgency of the data migration work is determined by the access frequency of the data stored in the extended storage. The workload monitoring module is responsible for collecting the access information of the workload to the data table;

[0043] The data migration job for the fields in the extended storage unit whose access frequency is higher than the custom threshold is given high priority; the fields in the extended storage unit whose access frequency is lower than the custom threshold are not migrated.

[0044] A modeless data storage system based on a hybrid storage structure that supports asynchronous changes. The methods for establishing, writing, and changing the mode of a data table are as follows:

[0045] Step S101: Create a data table

[0046] The metadata management module establishes the data model (metadata) information of the table. The data model is established by explicit declaration or implicit inference:

[0047] The explicit declaration method is to use the CREATE TABLE statement or other table creation commands supported by the system. Taking CREATE TABLE as an example, this statement requires the user to specify the name, type, and length of all fields in the table. The data model can be established based on the above information.

[0048] The implicit inference method is to establish the data model when the first write occurs through the INSERT statement or other data writing commands supported by the system. The implicit inference method infers the name, type and length of all fields from the data written for the first time, and establishes the data model based on the above information.

[0049] After establishing the data pattern, the storage module establishes a data storage structure; establishes corresponding fields in the basic storage unit for the data pattern according to the explicit declaration or implicit inference, and establishes an additional Extended field for storing a physical address pointer pointing to the corresponding record in the extended storage unit;

[0050] Allocate an empty file in the extended storage unit;

[0051] Step S102: Data writing

[0052] If the fields corresponding to the data subsequently written are all in the basic storage, the data will be stored in the basic storage unit;

[0053] If all or part of the fields corresponding to the subsequently written data are not in the basic storage unit, the unmatched parts will be stored in the extended storage unit, triggering a data mode change;

[0054] Step S103: Data mode change

[0055] Step S103.1: Add fields

[0056] If the written data contains a field that is not included in the current data model, the field is added to the current data model; the storage status information of the new field will be marked as "extended storage", and then the data migration process as described above is scheduled in combination with the information collected by the workload monitoring module;

[0057] Step S103.2: Change fields

[0058] If the field in the written data belongs to the current data mode, but the type does not match (for example, the field length does not match), the field type or length is increased to match the new data; the field is changed to update the field type or field length of the original field in the current data mode, and the storage status information of the field is marked as "mixed storage";

[0059] After the mark takes effect, the newly written data will be written to the extended storage; when reading, it will be read from the basic storage unit first, and if it cannot be read, it will be read from the extended storage unit;

[0060] Then, the data migration process as described above is performed in combination with the information collected by the workload monitoring module;

[0061] Step S103.3: Delete field

[0062] When the data cleaning module deletes a field, it only needs to delete the corresponding field metadata information in the data model.

[0063] The beneficial effects of the present invention are: the modeless data storage system that supports asynchronous changes based on a hybrid storage structure provides services to the outside through a mode storage system, and automatically triggers asynchronous mode changes when the written data does not match the current data mode, which can not only meet the high-performance access of massive data, but also realize flexible and free automatic data mode changes, thereby ensuring the performance and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Attached Figure 1 This is a schematic diagram of the architecture of a modeless data storage system that supports asynchronous changes based on a hybrid storage structure of the present invention.

[0066] Attached Figure 2 The figure is a schematic diagram of the data migration process of the present invention.

[0067] Attached Figure 3 This is a schematic diagram of the data writing process of the present invention.

[0068] Attached Figure 4 It is a schematic diagram of the data reading process of the present invention. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0070] The modeless data storage system based on hybrid storage structure supporting asynchronous changes includes a storage module, a metadata management module, a data reorganization module, a data cleaning module and a workload monitoring module;

[0071] The storage module includes two parts: a basic storage unit based on mode storage implementation and an extended storage unit based on modeless storage implementation;

[0072] In order to facilitate efficient addition, deletion or modification of fields, the basic storage unit adopts column storage. However, any storage system that supports mode storage is also applicable to the method proposed in the present invention.

[0073] The data in the extended storage unit is expressed in BSON or other common schema-free data formats.

[0074] The basic storage unit and the extended storage unit are associated with each other through a physical address pointer.

[0075] Although the data storage system provides modeless write and read services, the metadata of the system still contains the complete data mode information of the data table.

[0076] The data model recorded by the metadata management module contains field information of data in all tables, including field name, field type and field length;

[0077] Data schema information is obtained through explicit declaration (i.e., user declaration) or implicit inference (i.e., inference through written data).

[0078] The metadata management module is used to record the status information of each field, including storage status information and migration status information;

[0079] The storage status information is used to record the storage status of each field and determine the access mode of field data; the storage status of the field is divided into basic mode, extended mode and mixed mode;

[0080] Among them, the basic mode means that the corresponding field is stored in the basic storage unit, and the field is accessed through the basic storage unit;

[0081] The extended mode means that the corresponding fields are stored in the extended storage unit, and the fields are accessed through the extended storage unit;

[0082] The mixed mode means that the corresponding fields are stored in both the basic storage unit and the extended storage unit; when accessing, the fields are first accessed in the basic storage unit. If there is no corresponding data in the basic storage unit, the fields are accessed in the extended storage unit.

[0083] The migration status information is used to record the data migration process of each field to assist data migration; the migration status includes no migration process, migration started and migration in progress.

[0084] Since modeless storage requires more complex encoding and decoding when writing and reading, data (especially data that is frequently read and written) needs to be migrated from extended storage to basic storage as soon as possible.

[0085] The data reorganization module is responsible for migrating data from the extended storage unit to the basic storage unit; during the migration process, the system can still provide uninterrupted read and write services to the outside world;

[0086] The data migration process is as follows:

[0087] Step S1: Migration starts

[0088] Update the migration status information of the corresponding field in the metadata, marking it as "migrating";

[0089] Step S2: Create the migration target field

[0090] Allocate storage space for the fields that need to be migrated in the basic storage unit; for example, in column storage, create a new data file and allocate storage space for each field that needs to be migrated.

[0091] Step S3: Data conversion

[0092] Reading a data snapshot of the data to be migrated from the extended storage unit, parsing the corresponding fields to be migrated from the data snapshot, and storing them in the migration target field file established in the basic storage unit in step S2;

[0093] If the storage status information of the corresponding field in the data mode is "mixed mode" before the migration starts, the data is read from the old field in the basic storage unit and stored in the new field;

[0094] During the process of establishing the migration target field and data conversion, data access on the to-be-migrated field is still provided by the extended storage unit (in extended mode), or by the basic storage and extended storage jointly (in hybrid mode).

[0095] If there is data written after the data snapshot time point, it is appended to the basic storage unit through WAL (Write-ahead Log).

[0096] Step S4: Migration takes effect

[0097] Update the information of the field to be migrated in the data model of the metadata, marking it as being served by the basic storage unit rather than the extended storage unit; after the migration takes effect, data access on the migrated field is provided by the basic storage unit.

[0098] When the data model is changed, there will be unused data in the extension area. The data cleaning module is responsible for updating the BSON data in the extension area and removing the migrated fields;

[0099] The data cleaning module realizes data updating by creating a new record that does not contain useless data in the extended storage unit and pointing the pointer in the basic storage unit to the new record.

[0100] The urgency of the data migration work is determined by the access frequency of the data stored in the extended storage. The workload monitoring module is responsible for collecting the access information of the workload to the data table;

[0101] The data migration job for the fields in the extended storage unit whose access frequency is higher than the custom threshold is given high priority; the fields in the extended storage unit whose access frequency is lower than the custom threshold are not migrated.

[0102] The method of establishing, writing and changing the mode of the data table based on the hybrid storage structure and supporting asynchronous changes is as follows:

[0103] Step S101: Create a data table

[0104] The metadata management module establishes the data model (metadata) information of the table. The data model is established by explicit declaration or implicit inference:

[0105] The explicit declaration method is to use the CREATE TABLE statement or other table creation commands supported by the system. Taking CREATE TABLE as an example, this statement requires the user to specify the name, type, and length of all fields in the table. The data model can be established based on the above information.

[0106] The implicit inference method is to establish the data model when the first write occurs through the INSERT statement or other data writing commands supported by the system. The implicit inference method infers the name, type and length of all fields from the data written for the first time, and establishes the data model based on the above information.

[0107] After establishing the data pattern, the storage module establishes a data storage structure; establishes corresponding fields in the basic storage unit for the data pattern according to the explicit declaration or implicit inference, and establishes an additional Extended field for storing a physical address pointer pointing to the corresponding record in the extended storage unit;

[0108] Allocate an empty file in the extended storage unit;

[0109] Step S102: Data writing

[0110] If the fields corresponding to the data subsequently written are all in the basic storage, the data will be stored in the basic storage unit;

[0111] If all or part of the fields corresponding to the subsequently written data are not in the basic storage unit, the unmatched parts will be stored in the extended storage unit, triggering a data mode change;

[0112] Step S103: Data mode change

[0113] Step S103.1: Add fields

[0114] If the written data contains a field that is not included in the current data model, the field is added to the current data model; the storage status information of the new field will be marked as "extended storage", and then the data migration process as described above is scheduled in combination with the information collected by the workload monitoring module;

[0115] Step S103.2: Change fields

[0116] If the field in the written data belongs to the current data mode, but the type does not match (for example, the field length does not match), the field type or length is increased to match the new data; the field is changed to update the field type or field length of the original field in the current data mode, and the storage status information of the field is marked as "mixed storage";

[0117] After the mark takes effect, the newly written data will be written to the extended storage; when reading, it will be read from the basic storage unit first, and if it cannot be read, it will be read from the extended storage unit;

[0118] Then, the data migration process as described above is performed in combination with the information collected by the workload monitoring module;

[0119] Step S103.3: Delete field

[0120] When the data cleaning module deletes a field, it only needs to delete the corresponding field metadata information in the data model.

[0121] After adopting the modeless data storage system based on hybrid storage structure supporting asynchronous changes, in most cases, data access services are provided through mode storage. Compared with modeless storage, mode storage avoids the encoding and decoding work of formats such as BSON, and the access efficiency can be improved by 2-5 times compared with the data management system based on modeless storage. On the other hand, through asynchronous mode changes, the modeless data storage system based on hybrid storage structure supporting asynchronous changes can avoid the sudden drop in performance of the data management system based on mode storage during the mode change process, and provide more stable data access services.

[0122] In summary, the modeless data storage system that supports asynchronous changes based on a hybrid storage structure is a modeless storage system that is expanded through asynchronous mode updates based on a hybrid storage structure with and without modes, thereby being able to meet the high-performance storage and access of massive data and to achieve flexible and free automatic data mode changes;

[0123] At the same time, the system provides external services through a mode storage system, thus ensuring the performance and stability of the system;

[0124] On the other hand, when the written data does not match the current data mode, an asynchronous mode change will be triggered. During the change process, some data is served through modeless storage, thus avoiding performance fluctuations;

[0125] Asynchronous mode change jobs are scheduled through workload monitoring, and reasonable priorities are given to jobs based on data access frequency.

[0126] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A modeless data storage system based on a hybrid storage structure supporting asynchronous changes, characterized in that: It includes storage module, metadata management module, data reorganization module, data cleaning module and workload monitoring module; The storage module includes two parts: a basic storage unit based on mode storage implementation and an extended storage unit based on modeless storage implementation; The data model recorded by the metadata management module contains field information of data in all tables, including field name, field type and field length; Data schema information is obtained through explicit declaration or implicit inference; The metadata management module is used to record the status information of each field, including storage status information and migration status information; The data reorganization module is responsible for migrating data from the extended storage unit to the basic storage unit; During the migration process, the system can still provide uninterrupted read and write services to the outside world; The data cleaning module is responsible for updating the BSON data in the extension area and removing the migrated fields; The workload monitoring module is responsible for collecting the access information of the workload to the data table; Give high priority to data migration jobs for fields in the extended storage unit whose access frequency is higher than the custom threshold; Fields in the extended storage whose access frequency is lower than the custom threshold will not be migrated.

2. The modeless data storage system supporting asynchronous changes based on a hybrid storage structure according to claim 1, characterized in that: In order to facilitate efficient addition, deletion or modification of fields, the basic storage unit adopts column storage; The data in the extended storage unit is expressed in BSON format or a schema-free data format; The basic storage unit and the extended storage unit are associated with each other through a physical address pointer.

3. The modeless data storage system supporting asynchronous changes based on a hybrid storage structure according to claim 2, characterized in that: The storage status information is used to record the storage status of each field and determine the access method of the field data; The storage status of the field is divided into basic mode, extended mode and mixed mode; Among them, the basic mode means that the corresponding field is stored in the basic storage unit, and the field is accessed through the basic storage unit; The extended mode means that the corresponding fields are stored in the extended storage unit, and the fields are accessed through the extended storage unit; The mixed mode means that the corresponding fields are stored in both the basic storage unit and the extended storage unit; when accessing, the fields are first accessed in the basic storage unit. If there is no corresponding data in the basic storage unit, the fields are accessed in the extended storage unit.

4. The modeless data storage system supporting asynchronous changes based on a hybrid storage structure according to claim 1, characterized in that: The migration status information is used to record the data migration process of each field to assist data migration; The migration status includes No migration process, Migration started, and Migrating.

5. The modeless data storage system supporting asynchronous changes based on a hybrid storage structure according to claim 1, characterized in that: The process of completing data migration by the data reorganization module is as follows: Step S1: Migration starts Update the migration status information of the corresponding field in the metadata, marking it as "migrating"; Step S2: Create the migration target field Allocate storage space in the basic storage unit for the fields that need to be migrated; Step S3: Data conversion Reading a data snapshot of the data to be migrated from the extended storage unit, parsing the corresponding fields to be migrated from the data snapshot, and storing them in the migration target field file established in the basic storage unit in step S2; If the storage status information of the corresponding field in the data mode is a mixed mode before the migration starts, the data is read from the old field in the basic storage unit and stored in the new field; During the process of establishing the migration target field and data conversion, data access on the field to be migrated is still provided by the extended storage unit, or the basic storage and extended storage together; If there is data written after the data snapshot time point, it is appended to the basic storage unit through the write-ahead log WAL; Step S4: Migration takes effect Update the information of the fields to be migrated in the data model of the metadata, marking that they are served by the basic storage unit instead of the extended storage unit; After the migration takes effect, data access on the migrated fields is provided by the underlying storage unit.

6. The modeless data storage system supporting asynchronous changes based on a hybrid storage structure according to claim 1, characterized in that: The data cleaning module realizes data updating by creating a new record that does not contain useless data in the extended storage unit and pointing the pointer in the basic storage unit to the new record.

7. The modeless data storage system supporting asynchronous changes based on a hybrid storage structure according to any one of claims 1 to 6, characterized in that: The methods for creating, writing and changing the mode of a data table are as follows: Step S101: Create a data table The metadata management module establishes the data model information of the table. The data model is established by explicit declaration or implicit inference: The explicit declaration method is to create a table through the CREATE TABLE statement or other system-supported table creation commands; The implicit inference method is to establish the data model when the first write occurs through the INSERT statement or other data writing commands supported by the system. The implicit inference method infers the name, type and length of all fields from the data written for the first time, and establishes the data model based on the above information. After establishing the data pattern, the storage module establishes a data storage structure; establishes corresponding fields in the basic storage unit for the data pattern according to the explicit declaration or implicit inference, and establishes an additional Extended field for storing a physical address pointer pointing to the corresponding record in the extended storage unit; Allocate an empty file in the extended storage unit; Step S102: Data writing If the fields corresponding to the data subsequently written are all in the basic storage, the data will be stored in the basic storage unit; If all or part of the fields corresponding to the subsequently written data are not in the basic storage unit, the unmatched parts will be stored in the extended storage unit, triggering a data mode change; Step S103: Data mode change Step S103.1: Add fields If the written data contains fields that are not included in the current data mode, add the fields to the current data mode; The storage status information of the new field will be marked as "Extended Storage", and then the data migration process will be scheduled in combination with the information collected by the workload monitoring module; Step S103.2: Change fields If the field in the written data belongs to the current data mode, but the type does not match (for example, the field length does not match), then the field type or length is increased to match the new data; Changing a field updates the field type or field length of the original field in the current data mode, and marks the storage status information of the field as "mixed storage"; After the mark takes effect, the newly written data will be written to the extended storage; when reading, it will be read from the basic storage unit first, and if it cannot be read, it will be read from the extended storage unit; Then, the data migration process is scheduled based on the information collected by the workload monitoring module; Step S103.3: Delete field When the data cleaning module deletes a field, it only needs to delete the corresponding field metadata information in the data model.

8. The modeless data storage system supporting asynchronous changes based on a hybrid storage structure according to claim 7, characterized in that: In the explicit declaration method, the CREATE TABLE statement requires the user to specify the name, type, and length of all fields in the table and establish a data model based on the above information.

9. The modeless data storage system supporting asynchronous changes based on a hybrid storage structure according to claim 7, wherein The feature is that the implicit inference method infers the name, type and length of all fields from the data written for the first time. Create a data model based on the above information.

Images