A schema management method based on JSON document storage

Abstract

This application discloses a schema management method based on JSON document storage. First, schema extraction and discovery are performed, identifying all schemas within the JSON record. A hierarchical schema discovery algorithm groups equivalent record schemas into different categories. For a new record, its schema is compared with existing record schemas; if it's a new structure, it's immediately persisted. Next, schema storage is implemented, using an S-BucketTree data structure for schema discovery and storage to achieve real-time updates of the schema repository. Finally, schema querying is performed, implementing a query API based on the S-BucketTree data structure. Queries check whether a specified record schema has been persisted and whether the database contains a record schema with a specified attribute. This method effectively extracts all schemas from JSON records, improves the efficiency of real-time updates to the schema repository, and effectively supports basic queries such as schema existence queries and attribute existence queries.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology in shipbuilding, and in particular relates to a schema management method based on JSON document storage. Background Technology

[0002] In the actual processing of ship section manufacturing, ship manufacturing data from different sources is heterogeneous and of low quality, containing complex information and characterized by temporal continuity and real-time data requirements. Ship manufacturing data originates from various automation and information systems, and its storage and management are fundamental to ship manufacturing data analysis. Compared with traditional big data storage and analysis methods, the storage and mining of ship manufacturing data presents the following challenges: First, the data sources are complex, with different data structures and strong correlations between them; there is no unified method for constructing and managing heterogeneous data. Second, due to the massive volume of data, relational databases and data warehouses struggle to process and analyze massive amounts of industrial big data in real time. Third, ship manufacturing data requires multidimensional analysis because it originates from various aspects of the manufacturing process, and the relationships between these dimensions are extremely complex.

[0003] In recent years, the design principles of the Industrial Internet and smart factories have primarily focused on interoperability, virtualization, decentralization, distributed control and real-time functionality, service-oriented architecture, and rapid and easy maintenance. In the process of implementing massive industrial data storage and management, NoSQL databases holding semi-structured data have been widely adopted. Based on different data models, NoSQL databases are mainly divided into four categories: key-value, column-based, graph-based, and document-oriented. In particular, document-oriented databases offer flexible data models, extremely high query efficiency, are very easy to maintain, and provide rich APIs. However, without an explicit schema definition, retrieving the schema from a document store leads to a complex discovery process rather than a simple lookup. Summary of the Invention

[0004] The purpose of this invention is to solve the problems mentioned in the background art, and to propose a schema management method based on JSON document storage. This method uses a hierarchical schema discovery algorithm to assign recorded schemas from top to bottom to a category with an equivalent subtree. Furthermore, it proposes a data structure for schema discovery and storage, S-BucketTree (Schema BucketTree), to address the problem of real-time updates to the schema repository. Finally, it implements a query interface on S-BucketTree based on two basic queries.

[0005] To achieve the objectives of this invention, a schema management method based on JSON document storage is disclosed, comprising the following steps:

[0006] Step 1: Pattern Extraction and Discovery. Discover all patterns in the JSON record and group equivalent record patterns into different categories using a hierarchical pattern discovery algorithm. For a new record, compare its record pattern with the existing record patterns. If it is a new structure, persist it immediately.

[0007] Step 2, Schema storage, is achieved through real-time updates of the schema repository using the S-BucketTree data structure for schema discovery and storage;

[0008] Step 3: Pattern Query. The query API is implemented based on the S-BucketTree data structure. The query checks whether the specified record pattern has been persisted and whether the database contains the record pattern with the specified attribute.

[0009] Furthermore, the hierarchical pattern discovery algorithm assigns record patterns to corresponding buckets through a top-down, hierarchical equivalence identifier process. The output of this algorithm is an S-BucketTree, which compresses a class of equivalent record patterns into a single path. The S-BucketTree consists of a series of buckets, and each bucket has four variables: the first is an `id`, representing the class identifier, which is an ordered sequence; the second is an encoding map `CM`. b The first part consists of a tag-encoding pair; the second part is a flag, which indicates whether the path from this bucket to the root represents an equivalent record pattern; and the last part is a list of sub-buckets, which means that each sub-bucket of a bucket must have different IDs.

[0010] Furthermore, the specific process for pattern extraction and discovery in step 1 is as follows:

[0011] Step 1-1: Construct the record pattern S;

[0012] Step 1-2: Set the bucket to the current root node root_bucket, and the level l to 2;

[0013] Steps 1-3: If the bucket level is less than the maximum level of the recording pattern, perform a check on all nodes in the node set V(l) at level l according to the corresponding bucket's CM. b Encoding is performed; the detailed logic of the encoding node is: if CM b If a node has a label, replace the label code with the corresponding code; otherwise, assign a new code to the node and update the CM by adding this new label-code pair. b ;

[0014] Steps 1-4: Add the parent node's tag to the child node's tag;

[0015] Steps 1-5: Generate the IDs of the corresponding sub-buckets by sorting.

[0016] Steps 1-6: Record patterns are assigned to sub-buckets based on IDs. If a corresponding sub-bucket does not exist, a sub-bucket with the new ID is added to the sub-bucket list.

[0017] Steps 1-7: Generate the l-th level bucket of S-BucketTree, where the subtrees of the same root node are equivalent; set the bucket to sub_bucket, the level l to l+1, and repeat steps 1-3 to 1-7.

[0018] Steps 1-8: Mark the bucket with a flag, indicating that the path from this bucket to the root represents a class of equivalent record patterns.

[0019] Furthermore, the specific process for storing the schema in step 2 is as follows:

[0020] Step 2-1: For a newly inserted record, perform pattern extraction and discovery for each record in batch processing;

[0021] Step 2-2: If the pattern of this record is not persisted, represent a new path on the S-BucketTree as the new record pattern category; otherwise, the S-BucketTree will remain unchanged.

[0022] Furthermore, step 3, pattern query, includes two types: pattern existence query and attribute existence query. An API is proposed for each type of query, and an algorithm implementation based on S-BucketTree is given.

[0023] Furthermore, the specific process for querying the existence of a pattern is as follows:

[0024] Step 3-1-1: Construct the record pattern S;

[0025] Step 3-1-2: Set the current level l to 2;

[0026] Step 3-1-3: If the current level l is less than the maximum level of the recording pattern S, determine whether the label of the node at the current level is included in the encoding mapping of the corresponding bucket. If it is included, continue to execute downwards; otherwise, end the process.

[0027] Step 3-1-4: Determine whether the bucket contains a sub-bucket and whether the ID of the sub-bucket is the same as the sorted encoding sequence. If true, continue execution; otherwise, end the process.

[0028] Step 3-1-5: Set the bucket to sub_bucket, the level l to l+1, and repeat steps 3-1-3 to 3-1-4.

[0029] Step 3-1-6: If the flag of the last bucket is true, it means that the equivalent record pattern represented by the path from this bucket to the root has been persisted.

[0030] Furthermore, the specific process for attribute existence query is as follows:

[0031] Step 3-2-1: Determine the child node v of the root node in the attributes. l Check if the node is in the encoding mapping of root_bucket; if so, continue execution downwards.

[0032] Step 3-2-2, Determine v l Check if the encoding exists in the sub-bucket's id; if it does, continue execution.

[0033] Step 3-2-3, Determine v l child node v l+1 The tag and v l Check if the encoding combination exists in the encoding mapping of the corresponding sub-bucket; if so, continue execution downwards.

[0034] Step 3-2-4: Recursively determine the buckets containing the given attributes level by level;

[0035] Step 3-2-5: Create a bucket path representing the record pattern as the output.

[0036] Compared with the prior art, the significant advancements of this invention are: 1) proposing a hierarchical pattern discovery algorithm, which effectively extracts all patterns from JSON records by grouping equivalent patterns into corresponding categories; 2) proposing an S-BucketTree data structure for retrieving and storing patterns and supporting queries, thereby improving the efficiency of real-time updates to the pattern repository; 3) implementing a query interface on the pattern repository, enabling the S-BucketTree data structure to effectively support two basic queries.

[0037] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0039] Figure 1 This is an overall flowchart of a schema management method based on JSON document storage;

[0040] Figure 2It is a JSON record and its schema diagram based on a schema management method for storing JSON documents;

[0041] Figure 3 This is a schema management method based on JSON document storage to generate an example S-BucketTree diagram. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, Figure 1 The overall flowchart for a schema management method based on JSON document storage includes the following steps:

[0044] Step 1: Pattern Extraction and Discovery. Discover all patterns in the JSON record and group equivalent record patterns into different categories using a hierarchical pattern discovery algorithm. For a new record, compare its record pattern with the existing record patterns. If it is a new structure, persist it immediately.

[0045] Step 2, Schema storage, is achieved through real-time updates of the schema repository using the S-BucketTree data structure for schema discovery and storage;

[0046] Step 3, Schema Query: Based on the S-BucketTree data structure, the query API is implemented to check whether the specified record schema has been persisted and whether the database contains the record schema with the specified attribute. There are two types of schema queries: schema existence query and attribute existence query.

[0047] Example

[0048] This application proposes a pattern management method based on JSON document storage. A hierarchical pattern discovery algorithm is used to assign recorded patterns hierarchically from top to bottom to a category with an equivalent subtree. An S-BucketTree data structure for pattern discovery and storage is proposed to address the issue of real-time updates to the pattern repository. A query interface is effectively implemented on the S-BucketTree based on two basic queries. The relevant symbols are explained in Table 1.

[0049] Table 1. Explanation of Relevant Symbols

[0050]

[0051] According to JSON syntax, JSON objects are built on a collection of name-value pairs. Values ​​can be atomic values ​​(e.g., strings or numbers), another object, or an array. The structure of a JSON document can be represented as a tree, with its root node labeled "root".

[0052] In this invention, the record pattern S = (V, E) consists of a node set V and an edge set E. A node v ∈ V represents the corresponding node in the document, and its name is marked with the name of the corresponding node in the document. An edge e = root → v ∈ E represents an edge in the document that starts from the root node and leads to the specified node. An edge e = v1 → v2 ∈ E represents an edge that starts from v1 and leads to v2. In the edge e = v1 → v2, v1 is the parent node of v2, and v2 is the child node of v1. Figure 2 Four example record patterns are shown.

[0053] Two recording patterns S = (V, E) and S' = (V', E') are equivalent if and only if V = V' and E = E'. For example, Figure 2 S1 and S4 in the diagram are equivalent recording patterns.

[0054] In a record pattern, the root node's level is set to 1. For other nodes, their level is one level higher than their parent node's level. The set of nodes at level l is denoted as V(l). The maximum level of a record pattern is the maximum level of its leaf nodes, denoted as L. max .For example, Figure 2 The maximum level of S1 is 3.

[0055] In record mode, each path from the root node to a leaf node is called an attribute. For example... Figure 2 S1 in the table contains the following four attributes:

[0056] {root→task_id,root→worker→_id,root→worker→name,root→status}

[0057] When describing the same object type, the record patterns in a collection can also differ. Besides attribute sparsity, another major reason for different record patterns is attribute evolution, meaning that semantically equivalent attributes describing the same object type can have different formats. This is used to represent the equivalence of two sets of attributes. Attribute evolution includes two types: the first is naming conventions (i.e., semantically equivalent but with different labels), such as...

[0058]

[0059] The second type is structural variation (i.e., semantically equivalent but with different granularity), such as:

[0060]

[0061] {root→worker→name→first_name,

[0062] root→worker→name→last_name}

[0063] The pseudocode of the hierarchical pattern discovery algorithm described in step 1 is shown in Table 2. This method proposes a new hierarchical data structure, S-BucketTree (Schema Bucket Tree), whose path stores equivalent record pattern categories. Figure 3 Based on Figure 2 This is a documented example of an S-BucketTree. An S-BucketTree consists of a series of buckets, and each bucket in the S-BucketTree has four variables. The first is `id`, representing the class identifier, which is an ordered sequence. The second is the encoding map `CM`. b The first part consists of label-encoding pairs. The second part is a flag, which indicates whether the path from this bucket to the root represents an equivalent record pattern. The last part is a list of sub-buckets, meaning that each sub-bucket of a bucket must have distinct IDs.

[0064] Table 2 Hierarchical Pattern Detection Algorithm

[0065]

[0066] The specific process for pattern extraction and discovery in step 1 is as follows:

[0067] (1.1) Construct the record pattern S;

[0068] (1.2) Set the bucket to root_bucket and the level l to 2;

[0069] (1.3) If the bucket level is less than the maximum level of the recording pattern, for all nodes in V(l), according to the CM of the corresponding bucket... b Encode the nodes. The detailed logic for encoding nodes is: if CM b If a node has a label, replace the label code with the corresponding code; otherwise, assign a new code to the node and update the CM by adding this new label-code pair. b ;

[0070] (1.4) Add the parent node's label to the child node's label;

[0071] (1.5) Generate the IDs of the corresponding sub-buckets by sorting;

[0072] (1.6) Record patterns are assigned to sub-buckets based on id. If the corresponding sub-bucket does not exist, a sub-bucket with the new id is added to the sub-bucket list.

[0073] (1.7) Generate the l-th level bucket of the S-BucketTree, where subtrees of the same root node are equivalent. Set the bucket to sub_bucket, the level l to l+1, and repeat steps (1.3)-(1.7).

[0074] (1.8) Mark the bucket with a flag to indicate that the path from this bucket to the root represents a class of equivalent record patterns.

[0075] The algorithm based on hierarchical pattern discovery assigns record patterns to corresponding buckets using equivalence identifiers in a top-down, hierarchical manner. The output of this algorithm is an S-BucketTree, which compresses a class of equivalent record patterns into a single path.

[0076] The specific process of schema storage in step 2 is as follows:

[0077] (2.1) For a newly inserted record, perform the operation on each record in batch processing;

[0078] (2.2) If the pattern of this record is not persisted, a new path on the S-BucketTree is represented as the new record pattern category; otherwise, the S-BucketTree remains unchanged.

[0079] Step 3 describes two types of pattern queries: pattern existence query and attribute existence query. An API is provided for each type of query, along with an algorithm implementation based on S-BucketTree.

[0080] Schema existence queries are designed to check whether a specified record schema has been persisted. For example, performing a schema existence query as a pre-check can determine whether to insert a record into a collection with the same record schema, or to create a new collection for insertion. The API is:

[0081] SELECT S * from METADATA where S * =S(r)

[0082] Where S(r) is the record pattern for a given record r, and METADATA represents the record patterns persisted in all repositories. Duplicate record patterns are not persisted, therefore the query returns a boolean value, where true indicates existence.

[0083] The pseudocode for the schema-based query implementation on S-BucketTree is shown in Table 3:

[0084] Table 3 shows the existence of queries based on the S-BucketTree pattern.

[0085]

[0086] The specific process for querying the existence of a pattern in step 3 is as follows:

[0087] (3.1.1) Construct the record pattern S;

[0088] (3.1.2) Set the current level l to 2;

[0089] (3.1.3) If the current level l is less than the maximum level of the recording pattern S, determine whether the label of the node at the current level is included in the encoding mapping of the corresponding bucket. If it is included, continue to execute downwards; otherwise, end the process.

[0090] (3.1.4) Determine whether the bucket contains sub_bucket and whether the id of the sub-bucket is the same as the sorted encoding sequence. If true, continue to the next step; otherwise, end the process.

[0091] (3.1.5) Set the bucket to sub_bucket, the level l to l+1, and repeat steps (3.3)-(3.4) in a loop;

[0092] (3.1.6) If the flag of the last bucket is true, it means that the equivalent record pattern represented by the path from this bucket to the root has been persisted.

[0093] Attribute Existence Queries aim to determine record patterns containing specified attributes, providing finer-grained pre-checks by locating the attributes. The API is:

[0094] SELECT S from METADATA where attr∈S

[0095] Here, `attr` represents the given attribute. Due to variations in record patterns, a given attribute may exist in multiple record patterns; therefore, the query result is a set of record patterns. If the given attribute does not appear, the set is empty.

[0096] The pseudocode for attribute-based queries implemented on S-BucketTree is shown in Table 4:

[0097] Table 4 shows the attribute lookup based on S-BucketTree.

[0098]

[0099] The specific process for querying attribute existence in step 3 is as follows:

[0100] (3.2.1) Determine whether the v2 node of the attribute is in the encoding mapping of the root_bucket. If it exists, continue to execute the following steps.

[0101] (3.2.2) Determine if the encoding of v2 is in the id of the sub-bucket. If it exists, continue to the next step.

[0102] (3.2.3) Determine whether the combination of the label of v3 and the encoding of v2 is in the encoding mapping of the corresponding sub-bucket. If it exists, continue to execute the following steps.

[0103] (3.2.4) Recursively determine the bucket containing the given attributes level by level;

[0104] (3.2.5) Create a bucket path representing the record pattern as the output.

[0105] The implementation of the FindContainer method in Table 4 is shown in Table 5:

[0106] Table 5 Implementation of the FindContainer method

[0107]

[0108] Taking the attributes root→worker→name as an example, by iterating from top to bottom, we get that the flags of the two buckets in the third level of this S-BucketTree are both true, so the final result set contains the path from these two buckets to the root.

[0109] When the flag of the bucket containing the last node of the attribute is false. For example, suppose the attribute is root→status, the bucket level storing the last node is 2, and id = 1, 2, 3. The final result set is the paths from all its child nodes with flag set to true to the root node, because each node contains the given attribute. Therefore, it can be determined that the attribute root→status lies in one of the three record patterns.

[0110] When querying root→worker→_id, in the S-BucketTree, the last node belongs to a bucket with a depth of 3 and bucket id = 1,2. This bucket's flag is true, indicating that its path to the root node is included in the output. Furthermore, its child buckets do not have any extensions to the given attribute. Therefore, it can be determined that the record pattern contains root→worker→_id.

[0111] This application proposes a schema management method for JSON document storage, which can discover and persist JSON record schemas in a repository. During the discovery phase, a hierarchical schema discovery algorithm is used to group equivalent schemas, improving algorithm execution efficiency. A novel data structure, S-BucketTree, is proposed for schema storage, and two query interfaces are implemented based on this structure to assist developers in subsequent data analysis.

[0112] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0113] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A schema management method based on a JSON document store, characterized by, The method comprises the following steps: Step 1, mode extraction and discovery, discovering all modes in the JSON records, grouping equivalent record modes into different categories through a hierarchical mode discovery algorithm; For a new record, compare its record mode with the existing record modes, if it is a new structure, persist immediately; Step 2, mode storage, realizing real-time update of the mode storage through the S-BucketTree data structure for mode discovery and storage; Step 3, mode query, realizing the query API based on the S-BucketTree data structure, checking whether the specified record mode has been persisted through the query, and checking whether the record mode of the specified attribute is contained in the database The hierarchical pattern discovery algorithm assigns record patterns to corresponding buckets by a top-down equivalent identification, and the output of the algorithm is S-BucketTree, which compresses a class of equivalent record patterns into a path; S-BucketTree is composed of a series of buckets, and each bucket has four variables; the first is id, which represents the identifier of the class, and is an ordered sequence; the second is the encoding mapping CM b , which is composed of a label-encoding pair; the third is flag, which is used to show whether the path from the bucket to the root represents a class of equivalent record patterns; and the last is the sub-bucket list, that is, the sub-bucket sub_bucket of a bucket must have different ids. 2.The schema management method based on JSON document storage of claim 1, wherein, The specific process of the mode extraction and discovery in step 1 is as follows: Step 1-1, constructing a record mode S; Step 1-2, setting a bucket as a current root node root_bucket, and a level l as 2; Step 1-3, if the level of bucket is less than the maximum level of record schema, encode all the nodes in V(l) according to the CM of corresponding bucket b ; the detailed logic of encoding node is: if there is a label of some node in CM b , replace the label with the corresponding encoding; otherwise, assign a new encoding to the node and update CM b by adding this new label-encoding pair; Step 1-4, adding the label of the parent node in the label of the child node; Step 1-5, generating the id of the corresponding sub-bucket through sorting; Step 1-6, the record mode is allocated to the sub-bucket according to the id, if the corresponding sub-bucket does not exist, a sub-bucket with a new id is added in the sub-bucket list; Step 1-7, generating the l-level bucket of the S-BucketTree, wherein the sub-trees of the same root node are equivalent; setting the bucket as sub_bucket, and the level l as l+1, and executing steps 1-3 to 1-7 circularly; Step 1-8, marking the bucket through a flag, indicating that the path from the bucket to the root represents a class of equivalent record modes. 3.The schema management method based on JSON document storage of claim 1, wherein, The specific process of the mode storage in step 2 is as follows: Step 2-1, for a newly inserted record, executing the mode extraction and discovery on each record in a batch processing manner; Step 2-2, if the mode of the record has not been persisted, a new path on the S-BucketTree is represented as the new record mode category; Otherwise, the S-BucketTree remains unchanged. 4.The schema management method based on JSON document storage of claim 1, wherein, The mode query in step 3 comprises two kinds, namely, the mode existence query and the attribute existence query, for each kind of query, an API is proposed, and the algorithm implementation based on the S-BucketTree is given.

5. The schema management method based on a JSON document storage according to claim 4, characterized in that, The specific process of the mode existence query is as follows: Step 3-1-1, constructing a record mode S; Step 3-1-2, setting a current level l as 2; Step 3-1-3, if the current level l is less than the maximum level of the record mode S, judging whether the label of the current level node is contained in the encoding mapping of the corresponding bucket, if yes, continuing to execute downward, otherwise ending the process; Step 3-1-4, judging whether the bucket contains the sub_bucket, and whether the id of the sub-bucket is the same as the sorted encoding sequence, if yes, continuing to execute downward, otherwise ending the process; Step 3-1-5, setting the bucket as sub_bucket, and the level l as l+1, and executing steps 3-1-3 to 3-1-4 circularly; Step 3-1-6, if the flag of the last bucket is true, it means that the equivalent record schema represented by the path from this bucket to the root has been persisted. 6.The schema management method based on JSON document storage of claim 4, wherein, The specific process of attribute existence query is as follows: Step 3-2-1, judging whether the child node v of the root node root in the attribute l whether the node is in the encoding mapping of the root_bucket, if exists, continue to execute downwards; Step 3-2-2, judge whether the code of l is in the id of the sub-barrel, if it exists, continue to execute downwards; Step 3 - 2 - 3, judge v l whether the label of the child node v l+1 and the encoding of v l are combined in the encoding mapping of the corresponding child bucket, if it exists, continue to execute downward; Step 3-2-4, recursively determine the bucket containing the given attribute level by level; Step 3-2-5, establish the bucket path representing the record schema as the output.

Images