Load-oriented dynamic adaptive semi-structured JSON (JavaScript Object Notation) data relation mode representation method

Abstract

The invention discloses a load-oriented dynamic self-adaptive semi-structured JSON (JavaScript Object Notation) data relation mode representation method, which is characterized in that a system architecture comprising a storage layer and a query layer and a load-aware dynamic self-adaptive multi-granularity analysis method are adopted to realize JSON data access with low online analysis overhead; the storage layer comprises data storage and analysis, metadata storage, a mode analyzer, a column counter and a trigger; the query layer comprises a data initialization loading module, a mode checking module and a query execution module; the load-aware dynamic self-adaptive multi-granularity analysis comprises the steps of determining analysis granularity, counting the frequency of query access content, dynamically expanding the analysis granularity, analyzing the granularity and a dynamic self-adaptive strategy of schema. Compared with the prior art, the method has the advantages that the analysis granularity of the JSON data is dynamically adjusted, the overhead of data analysis and access in the query process is reduced, the query delay and the overall throughput of the system are improved, and efficient access of the JSON data is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of in-memory databases, and in particular to a relational model representation method for semi-structured JSON data oriented to dynamic load adaptation. Background Art

[0002] For the relational representation of JSON, researchers have proposed the Argo1 and Argo3 methods. These methods are divided according to the basic data types in JSON. Argo1 integrates the basic data types (String, Int, Bool) in JSON into multiple columns in the relational table, and assigns each row of data a unique row ID, and parses the nested structure to merge the array, and organizes the relationship between them by row ID. Argo3 organizes a data table for each data type, and then stores JSON in the corresponding table according to the relational representation of the data type. Some researchers have proposed the STDM and MTDM methods, which have the same principle as the Argo method. Some research work points out that a completely flat data representation is not necessarily the best. Therefore, based on the actual situation, the best way to represent data should be to change dynamically with the load or the hierarchical structure of the data.

[0003] Dynamically adaptive data management mode, online JSON relational model representation method that adapts to load changes

[0004] This method can not only adapt to load characteristics and maximize query efficiency, but also help balance storage resources and computing resources, which is reflected in reducing pre-parsing storage overhead and online parsing CPU overhead. Online dynamic maintenance of schema. Dynamically modifying its relational representation due to load changes can well adapt to changes in online application loads and ensure high query efficiency. However, further research is needed on which strategy to use for modification and how to represent the modified data.

[0005] At present, the relational representation methods of semi-structured JSON data include the following: single column storage (such as the introduction of JSON format data types such as jsonb), completely flat storage (such as Argo, STDM, MTDM), and adaptive methods (such as Sinew). Joy et al. proposed a method using a logical tiles abstraction layer, which hides the physical storage layout of the underlying data and realizes the layout transparency of the query processing component. The system also uses online data reorganization technology (on-line layout reorganization), which can dynamically optimize the physical storage layout of data according to the real-time changes of workload, so that the storage effect can be achieved close to the optimal without manual adjustment. For JSON relational representation and query, NoSQL non-relational databases such as MongoDB support loose data structures, store JSON data in bson (a new binary storage format) structure, and query data mainly in API, object-oriented query language, etc., and do not directly support SQL queries. Sinew discussed a system design that enables developers to continue to use self-describing formats to represent their JSON data without leaving SQL and traditional relational database systems. The system stores all JSON data in physical and virtual columns of a traditional relational database system and adds a layer on top of the database system to automatically provide users with dynamic relational views against which standard SQL queries can be performed.

[0006] Existing semi-structured JSON relational representation technologies usually have features such as non-fixed schema, self-description, and online data parsing. Because they support complex structures such as nested structures and arrays, they increase the diversity of their operations and enhance the expressiveness of data. In the rapidly developing Internet applications, this type of data interaction method and structure is very popular. Compared with relational databases, JSON data does not require a predefined schema, and its schema is not fixed. Therefore, when reading data, it is necessary to parse and judge its structure online, which brings additional overhead to the database or application. At the same time, because there is no constraint of a fixed schema, there may be data errors, as well as data inconsistencies caused by data writing and updating. For the above problems, the method of directly parsing all JSON has the problems of empty data redundancy and high parsing cost. After analyzing the existing benchmark and application load, the access to JSON data has a certain locality, that is, there is a certain hot spot phenomenon. Summary of the invention

[0007] The purpose of the present invention is to provide a load-oriented dynamically adaptive semi-structured JSON data relational model representation method in view of the deficiencies of the prior art. The system architecture including a storage layer and a query layer is adopted to adaptively pre-parse and manage the JSON structure according to the load characteristics and access frequency, and provide high query efficiency and low online parsing overhead to achieve efficient access to JSON data. The method introduces a pattern analyzer, a column counter and a trigger to adapt to the change of the load, and realizes the dynamic modification of the data parsing mode based on the statistical information of the access load; then, the trigger is a logical function abstraction for unified merging, and when the threshold is reached to trigger the unified merging, the data parsing and merging operations are completed; finally, in the query operation, the result data may need to be assembled into a JSON structure and returned to the user. The system maintains a multi-granularity mode of JSON. On the basis of ensuring data consistency, the schema with the coarsest parsing granularity is selected, so that the speed of assembling data is high, the method is simple, the use effect is good, and it has good application prospects and commercial development value.

[0008] The specific technical solution to achieve the purpose of the present invention is: a semi-structured load dynamic adaptive

[0009] The relational model representation method of JSON data is characterized in that the system architecture of the method includes a storage layer and a query layer. The storage layer includes: data storage and analysis, metadata, a model analyzer, a column counter and a trigger; the design of the storage layer mainly includes:

[0010] A1: JSON data is stored in the memory storage engine. In addition to data storage, its main functions include analyzing the data schema, designing new schemas based on the existing schema and column usage frequency statistics, and completing expansion operations (mapping the method for processing the schema in memory to triggers).

[0011] A2: Metadata (Catalog), which stores the meta-schema information and ownership relationships of the table;

[0012] A3: Schema Analyzer, whose main function is to analyze the schema of data, design new schemas based on the existing schemas and column usage frequency statistics, and complete expansion operations (map the methods for processing the schema in memory to triggers);

[0013] A4: Column Statistics (column data structure and statistics), a tool for counting JSON data. It counts the access frequency of the finest-grained structure for the existing JSON schema and the unparsed schema. At the same time, the structure maintains multiple counter structures (Counter), Map structures, and Bitmap structures to assist in completing schema updates, data parsing, data updates, and other operations.

[0014] A5: Triggers update the schema based on the behavior provided by the schema parser, and work with the column counter to parse and update the data.

[0015] The query layer mainly includes:

[0016] B1: Data initialization loader module, which provides data space initialization and data loading functions, including single write, batch import and other multi-granularity data writing capabilities;

[0017] B2: Lookup: This interface provides users with an interface to view the parsing mode of the current data.

[0018] B3: The query execution module is mainly responsible for executing the query submitted by the user, encapsulating the query results into the corresponding format, and returning them to the client.

[0019] During the storage and query process of JSON data, load-aware dynamic adaptive multi-granularity parsing includes:

[0020] C1: Count the sparsity of the key (key: num: total_num) in the JSON structure, consider load changes and access frequency, and determine the granularity of parsing based on the metadata;

[0021] C2: Count the frequency of query access content. For columns / keys with frequent access, parse them from the nested structure, store them separately as columns, and add a flag. The flag is used to identify the task status of background parsing historical data. After parsing, modify the flag and refresh the data in memory.

[0022] C3: Combining the above two types of metadata, dynamically expand the granularity of parsing through heuristic rules (setting thresholds). Since the granularity of parsing directly affects the parsing time and query response time, the load-aware dynamic expansion function can directly bring benefits to the parsing time and query time.

[0023] C4: The logical schema of data is maintained in memory. The schema contains the data parsing granularity, key information, and value type information. Based on the schema, data can be read dynamically and incremental parsing can be performed. The parsing granularity and schema can be dynamically adapted according to the query load and query statistics.

[0024] C5: Based on this adaptive strategy and combined with the column storage access characteristics, the goal is to maximize query latency.

[0025] Compared with the prior art, the present invention has the following significant beneficial effects:

[0026] 1) The present invention can dynamically adjust the parsing granularity of JSON data to reduce the overhead of data parsing and access during the query process, thereby improving the query latency and the overall throughput of the system.

[0027] 2) The present invention maintains a multi-granularity mode of JSON in the system. On the basis of ensuring data consistency, the schema with the coarsest parsing granularity is selected, which helps to minimize the data assembly overhead and improve the efficiency of data assembly.

[0028] 3) The present invention can adaptively pre-parse and manage JSON structure and data according to load characteristics and access frequency. The solution provides high query efficiency and low online parsing overhead to achieve efficient access to JSON data.

[0029] 4) The present invention introduces a pattern analyzer, a column counter and a trigger to adapt to changes in load.

[0030] Based on the statistical information of the access load, the analysis mode of dynamically modifying data is realized. The method is simple and easy to use.

[0031] The effect is good and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a system architecture diagram of the present invention;

[0033] Figure 2 This is a diagram of the parsing granularity and representation method of JSON data;

[0034] Figure 3 JSON multi-granularity data parsing flow chart;

[0035] Figure 4 Merge flow chart for JSON multi-granularity data parsing. DETAILED DESCRIPTION

[0036] This method is mainly designed for in-memory databases, but is also applicable to disk storage (page storage structure design and index structure acceleration). The present invention is further described in detail below through specific implementation.

[0037] Example 1

[0038] See also Figure 1 The system of the present invention includes: a storage layer and a query layer.

[0039] The storage layer mainly includes an in-memory storage engine (IN-MEMORY StorageEngine) for JSON data storage, metadata (Catalog), schema analyzer (Schema Analyzer), column statistics (ColumnStatistic, column data structure and statistical information) and triggers (Trigger). The metadata stores the meta-schema information, belonging relationships, etc. of the table; the main function of the schema analyzer is to analyze the schema of the data, design a new schema based on the frequency statistics of the existing schema and column usage, and complete the extension operation (map the method of processing the schema in memory to the trigger); the column statistics is a tool for counting JSON data, and counts the access frequency of the finest granular structure for the existing JSON schema and unparsed schema. At the same time, the structure maintains multiple counter structures (Counter), Map structures and Bitmap structures to assist in completing schema updates, data parsing, data updates and other operations; the trigger completes the schema update according to the behavior provided by the schema parser, and completes the parsing and updating of the data together with the column statistics.

[0040] The query layer includes user query execution (Query), data initialization loader (Loader) and mode viewing (Lookup) functions. The query function provides users with the function of querying and viewing JSON data. The data initialization loader provides the ability to write multi-granularity data such as batch import and single write. The mode view provides users with the ability to view the current parsed mode. In the query operation, it may be necessary to assemble the data into JSON and return the results to the user. The system maintains the multi-granularity mode of JSON. On the basis of ensuring data consistency, the schema with the coarsest parsing granularity is selected, so that the speed of assembling data is relatively fast. When some systems store JSON data, they will store the original JSON data, which will increase the storage overhead. This overhead is generally large, so the present invention does not do so.

[0041] Description of the functions of each component: The main function of the pattern analyzer is to analyze the schema of the data, design a new schema based on the existing schema and column usage frequency statistics, and complete the expansion operation (map the method of processing the schema in memory to the trigger). When the parsing mode of the data needs to be changed, the column identification bit (Modify bits) is maintained in the metadata. First, modify the identification bit, then process the column data, and after the processing is completed, change the data schema in the metadata. The column counter is a tool for counting JSON data column information. For the existing JSON schema and unparsed schema, it counts the access frequency of the most fine-grained structure. At the same time, multiple counter structures (Counter), Map structures, and Bitmap structures are maintained in this structure to assist in completing schema updates, data parsing, data updates, and other operations. A trigger is a logical function abstraction for unified merging. When the threshold is reached to trigger unified merging, the background thread adds a new column to the original table, parses the unparsed data and copies it to the newly added column, and copies the parsed data to the corresponding chunk of the newly added column. After confirming that the parsing is complete, the original column is deleted and the statistics of the column are cleaned up.

[0042] The following describes how to use the pattern analyzer, column counter, and trigger to adapt to changes in load and implement dynamic modification of data parsing patterns. Based on the statistical information of the access load, the access frequency of the key in the nested structure is recorded. When it is higher than the set threshold, the materialization operation is triggered. When it is higher than the set second gradient threshold, the trigger triggers a unified merge operation. In this process, the schema's key, data type, data frequency, data identifier and other meta information are maintained based on the Map data structure. The meta information includes parsing process metadata and statistical metadata. The meta information includes the JSON parsing meta information shown in Table 1 below for the parsing process:

[0043] Table 1 JSON parsing meta information

[0044] key type is_materialized from_chunk_id is_coalesce info#name object 1 100 0 info#kids#items#size int 1 100 0 info#kids#items array#string 1 100 0

[0045] Among them, "info#kids#items#size" is the meta information derived from the array type, which is used to identify the length of the array and assist in parsing and querying.

[0046] key: key name;

[0047] type: the type of data corresponding to the key;

[0048] is_materialized: Whether it is materialized, whether the field has been parsed, this field is used when the schema changes. When the data schema changes and conflicts with the previous data schema, to ensure the continued availability of the original data, the newly added data is not parsed into the new column;

[0049] from_chunk_id: The chunk id where parsing starts;

[0050] is_coalesce: Whether to trigger unified merging. Unified merging will parse the data before the chunk of this column (<from_chunk_id) and supplement it into this column.

[0051] Table 1 above describes the meta-information of the JSON parsing process. Among them, key is the key name; type is used to indicate the type of the data corresponding to the key; is_materialized indicates whether this column is materialized. This field identifies whether it has been parsed into a separate column. This field is used when the schema changes. When the data schema changes and conflicts with the previous data schema, to ensure the continued availability of the original data, the newly added data is no longer parsed into the new column. from_chunk_id is the chunk id where parsing starts; is_coalesce indicates whether to trigger unified merging. The purpose of the unified merging operation is to parse the data before the chunk of this column (<from_chunk_id) and supplement the data into this column so that all the data of this column can be accessed through the column name.

[0052] The statistical metadata is shown in the access frequency statistics data in Table 2 below:

[0053] Table 2 Access Frequency Statistics Data

[0054] key total_row frequency is_materialized info#name 100000 90000 0 info#kids#items 100000 85000 0

[0055] Among them, key: The key name;

[0056] total_row: The total number of data rows;

[0057] frequency: The access frequency;

[0058] is_materialized: Whether it is materialized.

[0059] Table 2 above records the statistical data such as the access frequency of the internal fields of the nested structure, mainly including the column name (key), the total number of data rows (total_row), the access frequency (frequency), and the identifier indicating whether this field is materialized (is_materialized).

[0060] See also Figure 2 The meta information in the JSON parsing process is the metadata and schema information of the JSON data relationship. Based on the above information, the parsing method and access path of the query data can be determined. The specific process is as follows: Figure 2 The case data in is used as an example to explain. Figure 2 The left side is the original JSON data. The outermost layer contains the key "info" whose value is an object type. To access the internal information of "info", you need to construct a path similar to "info / name" and parse its internal structure to access the inner data. Figure 2 The right side of the figure is the pre-parsed data. When the data is parsed to the outermost layer and expressed in the form of JSON outer layer kv, the inner layer data can be directly accessed without parsing the nested structure. Therefore, the statistical data in Table 2 can be combined to determine the data columns that need to be parsed when the parsing conditions are met, and parse the inner layer data to the outer layer. At the same time, the JSON outer layer data is organized and managed in the form of column storage. The column storage model is a storage method that organizes data by column. This model is particularly suitable for processing operations that require frequent access to a few columns in a large amount of data. It is common in online analytical processing (OLAP) systems. Column storage has made many optimizations for aggregate queries and statistical calculations under large amounts of data. As the load changes, different fields (virtual columns) under the nested structure are parsed to the outermost layer (physical column) at different time points. They are organized and managed separately in columns. This will cause the newly added columns to be misaligned in column storage, and the granularity will also be uneven.

[0061] See also Figure 3 , the figure shows the multi-granularity data parsing process, Col* represents the column, Chunk* represents the block structure in the column storage, the direction indicated by the red arrow in the figure is the direction of the block addition, and the brown and dark red arc arrows indicate the source of the data column, that is, Col4 is parsed from the nested structure of Col3. The parsing of the Col4 column is not triggered during Chunk1-Chunk3, and the column (Col4) does not exist; when the access frequency of the column reaches the threshold, a new data column Col4 is added to the block after Chunk4. Therefore, there will be a situation where the physical column Clo4 does not exist in Chunk1-Chunk3 (Col4 is a virtual column), and the physical column Col4 begins to appear after Chunk4. As the number of parseable columns increases, this situation will occur in many columns. This situation is described as: asymmetric column parsing granularity. Severe asymmetric column parsing granularity will bring the following problems: when parsing multiple newly added column data (non-original columns), the data parsing method and access path are not unified, and the data parsing method and access path are not unified. Figure 3For example, when accessing the data in Col4 (virtual column) from Chunk1 to Chunk3, the data must be parsed from Col3; in the blocks after Chunk4, data can be directly obtained from the column (physical column), which makes the access path complex and diverse, and also brings certain maintenance costs and parsing overhead. Therefore, consider introducing a unified merge mechanism, and when the dispersion of the column (the dispersion of the column indicates the proportion of non-parsed blocks of data, which is calculated from the metadata of the table data and access statistics) reaches the set threshold, the unified merge operation is triggered.

[0062] See also Figure 4 ,The multi-granularity data parsing and merging process is as follows: first, start the background ,merge thread to add a new column to the original table, and ,copy the unparsed block data to the newly added column, then, copy the parsed data to the corresponding Chunk of the ,newly added column. Finally, after confirming that the merge operation is completed, the original column is ,deleted, and the metadata such as the table column information is ,modified, and the statistical information of the column.

[0063] See also Figure 3-Figure 4 , the efficiency of data assembly. When the query filtering condition is the key or value in the inner nested structure of Col3, the filtering operation can be performed through Col4; then, when the query result needs to return the outer column data (Col1-Col2), the data can be directly obtained from other columns. However, there are also some cases where the data of the inner nested structure is returned. In this case, the data can be directly obtained from the Col3 column without the need to assemble through the parsed columns to obtain the returned results.

[0064] The query and storage of JSON data involve a query layer and a storage layer. The query layer is mainly responsible for query parsing and data processing; the main function of the storage layer is the memory mode storage and management of JSON data, which is mainly composed of table relational mode, metadata, mode analyzer, column counter, trigger and other modules. The method of directly representing JSON data in relational mode has the problems of empty data redundancy and high parsing cost. At the same time, there are a large number of nested structures in JSON data. When parsing data in deep structures, parsing overhead is required, but when all columns are pre-parsed, there is a problem of data redundancy. In addition, considering that the actual load has a certain access hotspot phenomenon. Therefore, the present invention proposes a load-oriented dynamic adaptive JSON data relational mode representation method, which aims to improve the effectiveness of JSON relational representation and query latency, and also improve the efficiency of data assembly in this process.

[0065] The above specific implementations are only for further explanation of the present invention, and are not intended to limit the patent of the present invention. All equivalent implementations of the present invention should be included in the scope of the claims of the patent of the present invention.

Claims

1. A method for representing semi-structured JSON data in a relational schema with dynamic load adaptation, characterized in that: A system architecture including a storage layer and a query layer and a load-aware dynamically adaptive multi-granularity parsing method are adopted to achieve JSON data access with low online parsing overhead. The storage layer includes: data storage and analysis, metadata storage, pattern analyzer, column counter and trigger; the query layer includes: data space initialization and data loading, pattern viewing and query execution; the load-aware dynamically adaptive multi-granularity parsing includes: determining the granularity of parsing, counting the frequency of query access content, dynamically expanding the granularity of parsing, and the dynamic adaptive strategy of parsing granularity and schema.

2. The method for representing relational schema of semi-structured JSON data with dynamic self-adaptation for load according to claim 1, characterized in that: The storage layer includes: A1: Store JSON data in the memory storage engine, design a new schema based on the existing schema and column usage frequency statistics, and complete the extended operation of mapping the method for processing the schema in memory to the trigger; A2: Metadata stores the meta-schema information and ownership relationships of the table; A3: The pattern analyzer designs a new schema based on the existing schema and column usage frequency statistics, and completes the extended operation of mapping the method for processing the pattern in memory to the trigger; A4: The column counter counts the access frequency of the most fine-grained structure for the existing JSON schema and the unparsed schema. At the same time, it maintains multiple counter structures, map structures, and bitmap structures to assist in completing schema updates, data parsing, and data updates. A5: The trigger completes the update of the schema according to the behavior provided by the schema parser, and completes the parsing and updating of the data together with the column counter.

3. The method for representing semi-structured JSON data in a relational model for dynamic load adaptation according to claim 1, characterized in that: The query layer includes: B1: Data initialization loading module, providing data space initialization and data loading, including single write, batch import and multi-granularity data write; B2: Mode viewing, providing an interface for users to view the parsing mode of the current data; B3: Query execution module, executes the query submitted by the user, encapsulates the query results into the corresponding format, and returns them to the client.

4. The method for representing relational schema of semi-structured JSON data with dynamic self-adaptation to load according to claim 1, characterized in that: The load-aware dynamic adaptive multi-granularity analysis includes: C1: Count the sparseness of the key key: num: total_num in the JSON structure, consider load changes and access frequency, and determine the granularity of parsing based on the metadata; C2: Count the frequency of query access content, parse the columns / keys with frequent access from the nested structure, store them separately as columns, and add a flag. The flag is used to identify the task status of the background parsing historical data. After parsing, modify the flag and refresh the data in the memory. C3: Combining the above two types of metadata, through heuristic rules, that is, setting thresholds, to achieve dynamic expansion The granularity of the analysis; C4: The logical schema of the data is maintained in memory. The schema contains the data parsing granularity, key information, and value type information. Based on the schema, data can be dynamically read and incremental parsing can be completed. The parsing granularity and schema can be dynamically adapted according to the query load and query statistics. C5: Based on the above adaptive strategy and combined with the column storage access characteristics, query latency is maximized.

Images