Column type partition storage method and device for variant type, equipment and medium

Abstract

The embodiment of the invention discloses a column type partition storage method and device for variant types, equipment and a medium. A specific embodiment of the method comprises the following steps: performing grammar embedding processing on a variant type; performing data extraction processing on the mode template statement; generating mode template metadata; performing data analysis processing on the initial semi-structured data; performing data grading matching processing on the initial key value pair data; performing conversion verification processing on each unmatched key value pair data and each composite key value pair data; performing data screening processing on each composite conversion sub-column; performing partition coding processing on each unmatched sub-column, each target conversion sub-column and each to-be-processed conversion sub-column; and performing determinant partition storage on the dynamic sub-column region data, the sparse column region data and the typed sub-column region data. According to the embodiment, waste of storage resources can be reduced.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more particularly to methods, apparatus, devices, and media for columnar partitioned storage of variant types. Background Technology

[0002] The Variant data type can store data of any structure and is often used to store semi-structured data. Currently, when using the Variant data type to store semi-structured data, the common approach is to directly store the semi-structured data as the raw value of the Variant field.

[0003] However, when using the Variant data type to store semi-structured data in the above manner, the following technical problems often arise: Using the Variant data type directly to store semi-structured data can easily lead to storing a large amount of redundant data such as duplicate key names or punctuation marks, resulting in a waste of storage resources.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide methods, apparatuses, electronic devices, and computer-readable media for columnar partitioned storage of variant types to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a columnar partitioned storage method for variant types. The method includes: responding to receiving initial semi-structured data and type definition syntax sent by a user client; performing syntax embedding processing on the variant type based on the type definition syntax to obtain a pattern template statement corresponding to the variant type; performing data extraction processing on the pattern template statement to obtain each template key-value pair data; generating pattern template metadata based on the each template key-value pair data; performing data parsing processing on the initial semi-structured data to obtain each initial key-value pair data; and performing data hierarchical matching processing on the each initial key-value pair data based on the pattern template metadata to obtain composite semi-structured data and each... The data consists of several unmatched key-value pairs, including the composite semi-structured data. The unmatched key-value pairs and the composite key-value pairs included in the composite semi-structured data are transformed and validated to obtain composite transformed sub-columns and unmatched sub-columns. Data filtering is performed on the composite transformed sub-columns to obtain target transformed sub-columns and unprocessed transformed sub-columns. Partitioning and encoding are performed on the unmatched sub-columns, target transformed sub-columns, and unprocessed transformed sub-columns to obtain dynamic sub-column area data, sparse column area data, and typed sub-column area data. The dynamic sub-column area data, sparse column area data, and typed sub-column area data are then stored in columnar partitioning.

[0008] Secondly, some embodiments of this disclosure provide a columnar partitioned storage device for variant types. The device includes: a syntax embedding unit configured to, in response to receiving initial semi-structured data and type definition syntax sent by a user client, perform syntax embedding processing on the variant type based on the type definition syntax to obtain a pattern template statement corresponding to the variant type; a data extraction unit configured to perform data extraction processing on the pattern template statement to obtain each template key-value pair data; a generation unit configured to generate pattern template metadata based on the each template key-value pair data; a data parsing unit configured to perform data parsing processing on the initial semi-structured data to obtain each initial key-value pair data; and a hierarchical matching unit configured to perform hierarchical matching processing on the each initial key-value pair data based on the pattern template metadata to obtain a composite semi-structured data. The system comprises: a semi-structured data set and unmatched key-value pairs, wherein the semi-structured data set includes composite key-value pairs; a transformation and verification unit configured to perform transformation and verification processing on the unmatched key-value pairs and the composite key-value pairs included in the semi-structured data set, to obtain composite transformation sub-columns and unmatched sub-columns; a data filtering unit configured to perform data filtering processing on the composite transformation sub-columns, to obtain target transformation sub-columns and unprocessed transformation sub-columns; a partitioning and encoding unit configured to perform partitioning and encoding processing on the unmatched sub-columns, target transformation sub-columns, and unprocessed transformation sub-columns, to obtain dynamic sub-column area data, sparse column area data, and typed sub-column area data; and a storage unit configured to perform columnar partitioning storage on the dynamic sub-column area data, the sparse column area data, and the typed sub-column area data.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above embodiments of this disclosure have the following beneficial effects: the columnar partitioning storage method for variant types in some embodiments of this disclosure can reduce storage resource waste. Specifically, the reason for storage resource waste is that directly using the Variant data type to store semi-structured data easily leads to the storage of a large amount of redundant data such as duplicate key names or punctuation marks, resulting in storage resource waste. Based on this, the columnar partitioning storage method for variant types in some embodiments of this disclosure firstly, in response to receiving initial semi-structured data and type definition syntax sent by the user client, performs syntax embedding processing on the variant type based on the type definition syntax to obtain the pattern template statement corresponding to the variant type. Thus, the variant type can be defined. Secondly, the pattern template statement is subjected to data extraction processing to obtain each template key-value pair data. Thus, each template key-value pair data can be obtained. Then, based on the above template key-value pair data, pattern template metadata is generated. Thus, pattern template metadata can be obtained. Then, the initial semi-structured data is subjected to data parsing processing to obtain each initial key-value pair data. Thus, the initial semi-structured data can be parsed to obtain the key-value pairs in the initial semi-structured data. Next, based on the aforementioned schema template metadata, hierarchical data matching is performed on each initial key-value pair, resulting in composite semi-structured data and each unmatched key-value pair. The composite semi-structured data includes each composite key-value pair. This determines whether each initial key-value pair matches the schema template metadata and filters out the mismatched initial key-value pairs. Then, transformation and verification processing is performed on the unmatched key-value pairs and the composite key-value pairs included in the composite semi-structured data, resulting in composite transformed sub-columns and unmatched sub-columns. This allows the unmatched key-value pairs and composite semi-structured data to be converted into columns. Then, data filtering processing is performed on the composite transformed sub-columns, resulting in target transformed sub-columns and unprocessed transformed sub-columns. This allows for further filtering of the composite transformed sub-columns, resulting in target transformed sub-columns and unprocessed transformed sub-columns. Next, the unmatched sub-columns, target transformed sub-columns, and unprocessed transformed sub-columns are partitioned and encoded to obtain dynamic sub-column data, sparse column data, and typed sub-column data. This allows for the determination of different storage partitions for different data. Finally, the dynamic sub-column data, sparse column data, and typed sub-column data are stored using columnar partitioning. This allows for partitioned storage of different data. Furthermore, because the semi-structured data can be converted into key-value pairs before storage, and then these key-value pairs are converted into columns, a large number of punctuation marks and duplicate key names can be removed from the semi-structured data, reducing storage resource waste. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of a columnar partitioned storage method for variant types according to the present disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of a columnar partitioned storage device for variant types according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1Flow 100 of some embodiments of the columnar partition storage method for variant types according to the present disclosure is shown. The columnar partition storage method for variant types includes the following steps: Step 101, in response to receiving the initial semi-structured data and type definition syntax sent by the user client, based on the type definition syntax, perform syntax embedding processing on the variant type to obtain a pattern template statement corresponding to the variant type.

[0021] In some embodiments, the execution entity (such as a computing device) of the columnar partition storage method for variant types may, in response to receiving the initial semi-structured data and type definition syntax sent by the user client, perform syntax embedding processing on the variant type based on the above type definition syntax to obtain the pattern template statement corresponding to the above variant type. Among them, the above execution entity may be a server. The above user client may be the client used by the user. The above initial semi-structured data may be the semi-structured data uploaded by the user. The format of the above initial semi-structured data may be, but is not limited to, JSON format or XML format. For example, the above initial semi-structured data may include each key-value pair. Each key-value pair in the above key-value pairs may include a field name and a field value. The field name included in the key-value pair may be the "key" of the key-value pair, that is, the unique identifier of the key-value pair. The field value included in the key-value pair may be the "value" of the key-value pair.

[0022] The above type definition syntax may be a syntax capable of declaring the above variant type. For example, the above type definition syntax may be "<user_id:INT 'User ID'>". The above variant type may be the Variant data type. The above pattern template statement may be a statement obtained by declaring the above variant type using the above type definition syntax.

[0023] In practice, the above execution entity may use a statement to declare a variable as the above variant type and declare the above type definition syntax to obtain a pattern template statement. Among them, the above statement to declare a variable may be a variable capable of declaring a data object.

[0024] For example, the above statement to declare a variable may be "DATA". As an example, when the statement to declare a variable is "DATA" and the type definition syntax is "<user_id:INT 'User ID'>, page.url:STRING 'Page URL'>", the pattern template statement may be "DATA VARIANT<user_id:INT 'User ID'>, page.url:STRING 'Page URL'".

[0025] Step 102, perform data extraction processing on the pattern template statement to obtain each template key-value pair data.

[0026] In some embodiments, the above-mentioned execution entity may perform data extraction processing on the above-mentioned pattern template statement to obtain respective template key-value pair data.

[0027] Among them, each template key-value pair data in the above-mentioned respective template key-value pair data may be a combination of each key-value pair extracted from the above-mentioned pattern template statement. Each template key-value pair data in the above-mentioned respective template key-value pair data may include a path field, a path pattern, a type field, a target type, a comment field, and a comment description.

[0028] The above-mentioned path field may be the name of a field in the template key-value pair data. For example, the above-mentioned path field may be "pattern". The above-mentioned path pattern may be the field value of the above-mentioned path field.

[0029] The above-mentioned type field may be the name of a field in the template key-value pair data for characterizing the data type. For example, the above-mentioned type field may be "type". The above-mentioned target type may be the field value of the above-mentioned type field.

[0030] The above-mentioned comment field may be the name of a field in the template key-value pair data for recording comments. For example, the above-mentioned comment field may be "comment". The above-mentioned comment description may be the field value of the above-mentioned comment field.

[0031] In practice, first, the above-mentioned execution entity may use a data extraction function to extract the definition string in the above-mentioned pattern template statement to obtain the extracted string as a template string. Among them, the above-mentioned definition string may be the string in the above-mentioned pattern template statement for defining the above-mentioned variant type. The above-mentioned data extraction function may be a function capable of extracting a specified substring from a string. For example, the above-mentioned data extraction function may be the REGEXP_SUBSTR function. As an example, when the pattern template statement is "CREATE TABLE user(data VARIANT<user_id:INT ‘user ID’>)", the definition string is "user_id:INT ‘user ID’", and the above-mentioned execution entity may use the following instruction statement: REGEXP_SUBSTR(CREATE TABLE user(‘data VARIANT<user_id:INT ‘user ID’>)’,‘<(. )>’,1,1,‘e’) to extract the definition string in the above-mentioned pattern template statement to obtain the template string "user_id:INT ‘user ID’".

[0032] Secondly, the template string can be split using a splitting function, with colons and spaces as delimiters, to obtain a sequence of substrings. This sequence of substrings can be used as the template substring sequence. The splitting function can be any function that splits a string into substrings based on a specified delimiter. For example, the split() function could be used.

[0033] Then, the first template substring in the above sequence can be identified as the path pattern. Next, the second template substring in the above sequence can be identified as the target type. Finally, the third template substring in the above sequence can be identified as a comment.

[0034] Next, the path field and path pattern described above can be combined into a key-value pair as the first key-value pair. For example, when the path field is "pattern" and the path pattern is "user_id", the first key-value pair could be "pattern:user_id". The type field and target type described above can be combined into a key-value pair as the second key-value pair. The comment field and comment description described above can be combined into a key-value pair as the third key-value pair.

[0035] Finally, the first key-value pair, the second key-value pair, and the third key-value pair can be combined into template key-value pair data.

[0036] Step 103: Generate schema template metadata based on each template key-value pair data.

[0037] In some embodiments, the execution entity can generate pattern template metadata based on the various template key-value pair data. The pattern template metadata can be data obtained by combining the various template key-value pair data. The pattern template metadata can include various field groups. Each field group can be template key-value pair data.

[0038] In some optional implementations of certain embodiments, the aforementioned execution entity can generate pattern template metadata based on the aforementioned template key-value pair data through the following steps: The first step is to perform the following steps for each template key-value pair in the above template key-value pair data: The first sub-step involves splitting the path pattern within the template key-value pair data into individual path strings based on a preset delimiter. The preset delimiter can be a pre-defined character, such as ".". Each path string can be a substring of the aforementioned path pattern. In practice, the executing entity can use the splitting function to divide the path pattern within the template key-value pair data into individual substrings using the preset delimiter.

[0039] The second sub-step involves performing string validation on each of the aforementioned path strings in response to determining that the path string meets a preset string condition, thereby obtaining a string validation result. The string condition may be that the path string does not contain wildcards. The string validation result may be a label indicating whether the path string conforms to a preset string format. The string validation result may be either "conforms" or "does not conform".

[0040] In practice, the aforementioned execution entity can use a data matching function to verify whether the path string conforms to the preset string format, and obtain the return value of the data matching function as the matching return value. The data matching function can be a function used to verify whether a string conforms to a specified format. For example, the data matching function could be the `re.match()` function. The preset string format can be a pre-defined format. For example, the preset string format could be "[a-zA-Z_][a-zA-Z0-9_]". The string is defined as follows: the first letter of the string is a character in the range "a" to "z" or "A" to "Z", and the characters in the string excluding the first letter are characters in the range "a" to "z", "A" to "Z", or "0" to "9". Then, in response to determining that the above match returns "None", "Does not match" can be determined as the string validation result. In response to determining that the above match returns "not none", "Matches" can be determined as the string validation result.

[0041] The third sub-step, in response to determining that the validation results of each obtained string meet the preset validation conditions, identifies the path strings that do not meet the above string conditions as the path strings to be processed. The above validation conditions can be that all the above string validation results are "compliant".

[0042] The fourth sub-step involves performing wildcard validation on each of the aforementioned path strings to be processed, obtaining various wildcard validation results. Each wildcard validation result can be a label used to indicate whether the path string to be processed meets the requirements. For example, a wildcard validation result can be "compliant" or "incompatible".

[0043] In practice, for each of the aforementioned path strings to be processed, in response to determining that the path string belongs to a preset string set, "matches" can be determined as the wildcard verification result for the corresponding path string. The preset string set can be a pre-defined set of strings that are allowed to appear in each path string to be processed. For example, the preset string set could be {" " In response to determining that the above-mentioned path string to be processed does not belong to the above-mentioned preset string set, "does not conform" can be determined as the wildcard verification result corresponding to the above-mentioned path string to be processed.

[0044] The fifth sub-step involves, in response to determining that the wildcard validation results satisfy a preset wildcard validation condition, performing type detection processing on the target types included in the template key-value pair data to obtain type detection results. The wildcard validation condition can be that all wildcard validation results are "compliant". The type detection results can be labels indicating whether the target types included in the template key-value pair data exist in a preset type table. For example, the type detection results can be "exist" or "not exist". The preset type table can be a table containing various data types. The data types included in the preset type table can include, but are not limited to: int, float, and char.

[0045] In practice, a data lookup function can be used to search for the target type included in the template key-value pair data from the aforementioned preset type table, and the return value of the data lookup function is used as the lookup return value. The data lookup function can be a function capable of searching for specified data. For example, the data lookup function can be the `find()` function. The lookup return value can be used to indicate whether the data lookup function has found a value of the target type included in the template key-value pair data from the aforementioned preset type table. For example, the lookup return value can be "-1", indicating that it was not found, or it can be the index of a string of the same target type in the aforementioned preset type table. Then, in response to determining that the lookup return value indicates that it was not found, "not found" can be determined as the type detection result. As an example, when the lookup return value is "-1", it can be determined that the lookup return value indicates that it was not found. Then, in response to determining that the lookup return value cannot indicate that it was not found, "existent" can be determined as the type detection result.

[0046] The second step involves, in response to the determination that the obtained detection results of each type satisfy the preset type detection conditions, performing conflict detection processing on each path pattern included in the template key-value pair data to obtain conflict detection results. The type detection conditions can be that all the above type detection results are "existing". The conflict detection results can be labels used to characterize whether there is a conflict between the various path patterns. For example, the conflict detection results can be "conflict exists" or "no conflict exists".

[0047] The third step involves integrating the template key-value pairs in response to the determination that the conflict detection results meet the preset conflict detection conditions, thereby obtaining the schema template metadata. The conflict detection condition can be that the conflict detection result is "no conflict exists." In practice, each template key-value pair can be defined as a field group. Then, these defined field groups can be combined to form the schema template metadata.

[0048] In some optional implementations of certain embodiments, the aforementioned execution entity can perform conflict detection processing on the various path patterns included in the aforementioned template key-value pair data through the following steps to obtain conflict detection results: The first step is to perform the following steps for each path pattern included in the above template key-value pair data: The first sub-step is to determine the above path pattern as the path pattern to be matched.

[0049] The second sub-step involves identifying each path pattern among the aforementioned path patterns that meets preset pattern conditions as a path pattern to be processed. The aforementioned pattern conditions can include the path pattern not being one of the path patterns to be matched.

[0050] The third sub-step involves performing a data comparison process between each of the aforementioned path patterns to be processed and the path pattern to be matched, thereby obtaining a path comparison result. This path comparison result can be a label used to characterize whether the path pattern to be processed and the path pattern to be matched are completely identical. For example, the path comparison result can be "TRUE" or "FLASE".

[0051] In practice, for each of the aforementioned path patterns to be processed, the executing entity can input both the path pattern to be processed and the path pattern to be matched into a data comparison function, and obtain the return value of the data comparison function as the path comparison result. The data comparison function can be a function used to compare whether two strings are the same. For example, the data comparison function could be the EXACT function.

[0052] The second step involves determining the conflict information as a conflict detection result in response to the determination that each path alignment result meets a preset conflict condition. The conflict condition can be that any one of the path alignment results is "TRUE". The conflict information can be information used to characterize the existence of a conflict between the various path patterns. For example, the conflict information can be "a conflict exists".

[0053] Optionally, after determining that the obtained path comparison results meet the preset conflict conditions and identifying the preset conflict information as the conflict detection result, the execution entity may further perform the following steps: In response to the determination that the obtained path alignment results do not meet the above conflict conditions, the following steps are performed: The first step involves performing character segmentation on each path pattern within the template key-value pair data, resulting in a sequence of path segmentation strings. Each path segmentation string in this sequence can be a substring obtained by segmenting the path pattern. In practice, for each path pattern within the template key-value pair data, the execution entity can use a preset path separator as the separator and the aforementioned splitting function to perform character segmentation on the path pattern, obtaining individual path segmentation strings. The preset path separator can be ".". Then, the path segmentation strings can be arranged into a sequence according to their order within the path patterns.

[0054] The second step is to identify the path segmentation strings that meet the preset data conditions from the obtained path segmentation string sequences as the first-level segmentation strings. The preset data condition can be that the path segmentation string is the first one in the path segmentation string sequence.

[0055] Third, for each of the above first-level split strings, perform the following steps: The first sub-step is to determine the above-mentioned first-level segmentation string as the target first-level string.

[0056] The second sub-step involves identifying each of the aforementioned first-level segmentation strings that meets the preset data matching conditions as the respective first-level strings to be processed. The data matching condition can be that the first-level segmentation string is not the target first-level string.

[0057] The third sub-step involves comparing each of the aforementioned primary strings with the target primary string to obtain a primary comparison result. This primary comparison result can be a label indicating whether the primary string and the target primary string are completely identical. For example, the primary comparison result could be "TRUE" or "FLASE".

[0058] In practice, for each of the above-mentioned first-level strings to be processed, the execution entity can input the first-level string to be processed and the target first-level string into the data comparison function, and obtain the return value of the data comparison function as the first-level comparison result.

[0059] The fourth sub-step involves determining, in response to the determination that each obtained first-level alignment result meets a preset first-level alignment condition, the preset non-conflict information is identified as the conflict detection result. Here, the first-level alignment condition can be that each of the above first-level alignment results is "FLASE". The non-conflict information can be information used to characterize that there is no conflict between the above path patterns. For example, the non-conflict information can be "no conflict".

[0060] The fifth sub-step involves determining that each of the obtained first-level alignment results does not meet the aforementioned first-level alignment conditions, and then identifying each of the aforementioned first-level alignment results that meets the preset first-level result conditions as the respective target first-level alignment results. Here, the aforementioned first-level result conditions can be that the first-level alignment result is "TRUE".

[0061] The sixth sub-step involves determining the path patterns corresponding to the first-level comparison results of each target as the respective first-level path patterns.

[0062] The seventh sub-step is to determine the target types corresponding to the above-mentioned first-level path patterns as the respective first-level target types.

[0063] The eighth sub-step involves determining, in response to the fact that none of the aforementioned primary target types meet the preset type-same condition, the conflict information is identified as a conflict detection result. The type-same condition can be that all the aforementioned primary target types are identical.

[0064] Step 104: Perform data parsing processing on the initial semi-structured data to obtain each initial key-value pair data.

[0065] In some embodiments, the execution entity may perform data parsing processing on the initial semi-structured data to obtain various initial key-value pair data. Each initial key-value pair data may be a key-value pair extracted from the initial semi-structured data. Each initial key-value pair data may include a field name and a field value.

[0066] In practice, the aforementioned executing entity can use a data parsing tool to convert the initial semi-structured data into a structured table as the initial structured table. This structured table can be used to record the key-value pairs in the initial semi-structured data. The data parsing tool can be any tool capable of converting semi-structured data into a structured table. For example, the data parsing tool can be an ETL (Extract-Transform-Load) tool.

[0067] Then, the data in the first row of the initial structured table can be identified as field names. Then, for each field name, the data in the same column as that field name in the initial structured table can be identified as the corresponding field value. Then, for each field value, the field value and its corresponding field name can be combined into a key-value pair as the initial key-value pair data.

[0068] Step 105: Based on the schema template metadata, perform hierarchical matching processing on each initial key-value pair to obtain composite semi-structured data and each unmatched key-value pair.

[0069] In some embodiments, the execution entity may perform hierarchical matching processing on the initial key-value pairs based on the pattern template metadata to obtain composite semi-structured data and unmatched key-value pairs. The composite semi-structured data may be data obtained by combining the initial key-value pairs that match the pattern template metadata. The composite semi-structured data may include composite key-value pairs. Each composite key-value pair may be an initial key-value pair that matches the pattern template metadata. Each unmatched key-value pair may be an initial key-value pair that does not match the pattern template metadata.

[0070] In some optional implementations of certain embodiments, the execution entity may perform hierarchical matching processing on the initial key-value pairs based on the pattern template metadata to obtain composite semi-structured data and each unmatched key-value pair: The first step is to perform the following steps for each initial key-value pair in the above initial key-value pair data: The first sub-step involves determining the field names and field values ​​included in the aforementioned initial key-value pair data as the initial field names and initial field values, respectively.

[0071] The second sub-step involves determining each path field in each field group of the aforementioned pattern template metadata as the respective template path field.

[0072] The third sub-step is to determine the path patterns corresponding to the above-mentioned template path fields as the values ​​of the template path fields.

[0073] The fourth sub-step involves searching for the template path field value corresponding to the initial field name from each of the aforementioned template path field values, thus obtaining the field value search result. This field value search result can be a label indicating whether a template path field value with the same initial field name exists among the aforementioned template path field values. For example, the field value search result can be "exists" or "does not exist".

[0074] In practice, the aforementioned execution entity can use the aforementioned data lookup function to search for template path field values ​​with the same initial field name from the various template path field values, obtaining the return value of the aforementioned data lookup function as the lookup field return value. For example, the aforementioned lookup field return value can be "-1", indicating that it was not found, or it can be the index of the template path field value with the same initial field name among the various template path field values. Then, in response to determining that the aforementioned lookup field return value indicates that it was not found, "not found" can be determined as the field value lookup result. As an example, when the aforementioned lookup field return value is "-1", it can be determined that the aforementioned lookup field return value indicates that it was not found. Then, in response to determining that the aforementioned lookup field return value cannot indicate that it was not found, "exists" can be determined as the field value lookup result. As an example, when the lookup field return value is the index of the template path field value with the same initial field name among the various template path field values, it can be determined that the aforementioned lookup field return value cannot indicate that it was not found.

[0075] The fifth sub-step involves determining that the search result for the aforementioned field value does not meet the preset search criteria, and classifying the initial key-value pair data as unmatched key-value pair data. The search criteria can be that the search result for the aforementioned field value is "exists".

[0076] The sixth sub-step involves determining the template path field value that matches the initial field name among the aforementioned template path field values, in response to the determination that the search results satisfy the search conditions. In practice, the template path field value with the same initial field name among the aforementioned template path field values ​​can be determined as the target template path field value.

[0077] The seventh sub-step is to determine the target type of the type field corresponding to the target template path field value as the target type field value.

[0078] The eighth sub-step involves performing data transformation on the initial field value based on the target type field value to obtain the transformed field value. The transformed field value can be the data obtained by converting the initial field value to the data type of the target type field value.

[0079] In practice, the aforementioned execution entity can use a type casting method to forcibly convert the initial field value to the data type corresponding to the target type field value, obtaining the converted initial field value as the conversion field value. For example, when the target type field value is "INT" and the initial field value is 3.23, the initial field value can be forcibly converted to the int type using the statement "(int)3.23", obtaining the converted data 3 as the conversion field value.

[0080] The ninth sub-step involves combining the initial field name, the target type field value, and the transformation field value into composite key-value pairs.

[0081] The second step is to generate composite semi-structured data based on the obtained composite key-value pairs. In practice, the aforementioned composite key-value pairs can be combined into composite semi-structured data.

[0082] Step 106: Perform transformation and verification processing on each unmatched key-value pair data and each composite key-value pair data included in the composite semi-structured data to obtain each composite transformed sub-column and each unmatched sub-column.

[0083] In some embodiments, the execution entity may perform transformation and verification processing on the unmatched key-value pair data and the composite key-value pair data included in the composite semi-structured data to obtain each composite transformed sub-column and each unmatched sub-column.

[0084] Each of the aforementioned composite transformation sub-columns can be a column in a preset data table used to store composite key-value pair data. Each of the aforementioned composite transformation sub-columns includes a field name and a field value. Each field value in each composite transformation sub-column can be a transformed field value in the composite key-value pair data.

[0085] Each of the aforementioned unmatched sub-columns can be a column in a preset data table used to store unmatched key-value pairs. Each of the aforementioned unmatched sub-columns can include a field name and individual field values. Each field value in each unmatched sub-column can be a field value from the unmatched key-value pair data.

[0086] In practice, for each composite key-value pair within the aforementioned composite semi-structured data, firstly, the initial field names and target type field values ​​of the composite key-value pair can be combined into table information. Then, a set construction function can be used to combine the obtained table information into a set of table information. This set construction function can be the `set()` function.

[0087] Then, for each table in the aforementioned table information set, the initial field names included in the table information can be determined as column field names. The target type field values ​​included in the table information can be determined as table data types. Then, a column with the aforementioned column field names and the aforementioned table data type can be created in a preset data table as an initial column. This preset data table can be a data table pre-generated in the database. The field name of the initial column can be the name of the initial column, that is, the first row of data in the initial column. As an example, when the preset data table is named "sheet", the column field name is "user_id", and the table data type is "INT", the initial column can be created using the following SQL statement: "ALTER TABLE sheet ADD COLUMN user_id INT". Then, based on the obtained initial columns, the following update steps are performed: For each composite key-value pair in the composite semi-structured data, firstly, the initial column corresponding to the composite key-value pair in each initial column can be determined as the target initial column. As an example, the initial columns whose field names are the same as the initial field names in the composite key-value pair data, and whose data types are the same as the target type field values ​​in the composite key-value pair data, can be determined as the target initial columns. Then, the transformed field values ​​included in the composite key-value pair data can be stored in the target initial columns as their field values ​​to update them, resulting in the updated target initial columns. Finally, the updated target initial columns can be used as the respective composite transformation sub-columns.

[0088] Next, for each of the aforementioned unmatched key-value pairs, firstly, the field names and values ​​included in the unmatched key-value pair data can be combined into unmatched table information. Then, the aforementioned set construction function can be used to combine the obtained table information into an unmatched table information set.

[0089] Then, for each unmatched table information in the aforementioned unmatched table information set, the field names included in the unmatched table information can be determined as the target column field names. Then, a column with the aforementioned target column field names can be created in a preset data table as an initial unmatched column. The creation method can be referred to the specific implementation of step 106, and will not be repeated here.

[0090] Then, based on the obtained initial unmatched columns, the following steps are performed: For each unmatched key-value pair in the aforementioned unmatched key-value pair data, firstly, the initial unmatched column whose field name is the same as the field name included in the aforementioned unmatched key-value pair data is determined as the target initial unmatched column. Secondly, the field values ​​included in the aforementioned unmatched key-value pair data are added to the target initial unmatched column as field values ​​to update the target initial unmatched column, resulting in the updated target initial unmatched column. Finally, the updated target initial unmatched columns are determined as unmatched sub-columns.

[0091] Step 107: Perform data filtering on each composite transformation sub-column to obtain each target transformation sub-column and each transformation sub-column to be processed.

[0092] In some embodiments, the execution entity may perform data filtering on the aforementioned composite transformation sub-columns to obtain target transformation sub-columns and transformation sub-columns to be processed. Each target transformation sub-column may be a composite transformation sub-column that satisfies preset sub-column conditions. Each transformation sub-column to be processed may be a composite transformation sub-column that does not satisfy the preset sub-column conditions.

[0093] In some optional implementations of certain embodiments, the execution entity may perform data filtering processing on the aforementioned composite transformation sub-columns through the following steps to obtain each target transformation sub-column and each transformation sub-column to be processed: For each of the above composite transformation sub-columns, perform the following steps: The first step involves determining that the composite transformation sub-column meets a preset sub-column condition, and then designating it as the target transformation sub-column. This sub-column condition can be defined as the correspondence between the values ​​of each field in the composite transformation sub-column and its data type. In practice, for each field value in the composite transformation sub-column, the field value and the data type of the composite transformation sub-column can be input into a type determination function, and the return value of the type determination function can be used as the type return value. This type determination function can be a function used to determine whether a string is a specified data type. For example, the type determination function could be the `isinstance()` function. The type return value can be "True" or "False". Then, in response to determining that all obtained type return values ​​are "True", it can be determined that the composite transformation sub-column meets the aforementioned sub-column condition, and thus, it can be designated as the target transformation sub-column.

[0094] The second step involves determining that the composite transformation sub-column does not meet the aforementioned sub-column conditions, and then designating it as a transformation sub-column to be processed. In practice, if any of the obtained return values ​​of various types is determined to be "Flase", it can be determined that the composite transformation sub-column does not meet the aforementioned sub-column conditions, and thus it can be designated as a transformation sub-column to be processed. The method for generating the return values ​​of various types based on the composite transformation sub-column can be found in the specific implementation of the first step in step 107, and will not be repeated here.

[0095] Step 108: Perform partitioning and encoding on each unmatched sub-column, each target transformation sub-column, and each transformation sub-column to be processed to obtain dynamic sub-column area data, sparse column area data, and typed sub-column area data.

[0096] In some embodiments, the execution entity may perform partitioning and encoding on the unmatched sub-columns, the target transformation sub-columns, and the transformation sub-columns to be processed to obtain dynamic sub-column area data, sparse sub-column area data, and typed sub-column area data. The dynamic sub-column area data may be data obtained by combining the unmatched sub-columns. The sparse sub-column area data may be data obtained after processing the transformation sub-columns to be processed. The sparse sub-column area data may include various sparse data. Each sparse data may be data obtained after processing the field values ​​in the transformation sub-column to be processed. The typed sub-column area data may be data obtained after processing the target transformation sub-columns. The typed sub-column area data may include various typed key-value pairs. Each typed key-value pair may be data obtained after processing the field values ​​in the target transformation sub-column. Each typed key-value pair may include a target field name and target field feature information. The target field name may be the field name included in the target transformation sub-column. The aforementioned target field feature information can be data obtained by encoding or data transformation of the field values ​​included in the target transformation sub-column.

[0097] In addressing the aforementioned technical challenges by employing technical solutions, the application scenario—SaaS multi-tenant services—often presents the following technical issues: Variant cannot recognize the specific structure and content characteristics of semi-structured data, resulting in the inability to perform differentiated compression on different data types during storage, leading to a large storage footprint. Considering the following requirements for this application scenario: In SaaS multi-tenant services, multiple enterprises often share a database. Due to limited database storage space, it is necessary to compress the storage space occupied by the data as much as possible to avoid data loss and ensure data integrity. Therefore, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may perform partitioning and encoding processing on the unmatched sub-columns, the target transformed sub-columns, and the transformed sub-columns to be processed through the following steps to obtain dynamic sub-column area data, sparse sub-column area data, and typed sub-column area data: The first step is to combine the above unmatched sub-columns into dynamic sub-column data.

[0098] The second step is to determine the values ​​of each field included in each of the above target transformation sub-columns as the values ​​of each target field.

[0099] Third, for each of the target field values ​​mentioned above, perform the following steps: The first sub-step involves, in response to determining that the target field value meets a preset discrete data condition, performing numerical encoding on the target field value to obtain target field encoded data as target field feature information. The target field encoded data can be a feature vector obtained after encoding the target field value. The discrete data condition can be that the target field value is a string and does not meet a preset time character condition. The time character condition can be that the string conforms to a preset time format. The preset time format can be a format containing year, month, day, hour, minute, and second. For example, the preset time format can be "%Y-%m-%d %H:%M:%S". As an example, "2025-10-25 12:45:12" conforms to the preset time format. In practice, the executing entity can input the target field value and the preset time format into a time determination function, obtaining the return value of the time determination function as the time return value. The time determination function can be a function capable of determining whether the input string conforms to the preset time format. For example, the time determination function can be the strptime() function. The aforementioned time return value can be either "ValueError" or the aforementioned target field value. Then, in response to determining that the aforementioned time return value is "ValueError", it can be determined that the aforementioned target field value does not meet the aforementioned time character condition. In practice, in response to determining that the aforementioned target field value meets the aforementioned discrete data condition, the aforementioned execution entity can perform one-hot encoding on the aforementioned target field value to perform numerical encoding processing on the aforementioned target field value, obtaining the target field encoded data as the target field feature information.

[0100] The second sub-step involves determining the target field value as target field feature information in response to the determination that the target field value meets a preset continuous data condition. The continuous data condition can be that the target field value is a numerical value.

[0101] The third sub-step involves, in response to determining that the target field value meets a preset time data condition, performing a time feature transformation on the target field value to obtain target time data as target field feature information. Here, the aforementioned time data condition can be that the target field value is a string and meets the aforementioned time character condition. The aforementioned target time data can be a timestamp of the target field value.

[0102] In practice, the aforementioned execution entity can generate a time return value based on the target field value and the preset time format. The method for generating the time return value based on the target field value and the preset time format can be found in the specific implementation of step 108, and will not be repeated here. In response to determining that the time return value is not "ValueError", it can be determined that the target field value meets the aforementioned time character condition. Then, in response to determining that the target field value meets the aforementioned time data condition, the target field value can be input into a time conversion function to obtain the timestamp of the target field value as the target time data. The aforementioned time conversion function can be a function capable of converting the time represented by a struct tm structure into a timestamp. For example, the aforementioned time conversion function can be the mktime() function. Then, the aforementioned target time data can be determined as the target field feature information.

[0103] The fourth sub-step is to determine the field name corresponding to the above target field value as the target field name.

[0104] The fifth sub-step involves combining the target field names and the obtained target field feature information into typed key-value pair data.

[0105] The fourth step is to combine the obtained typed key-value pairs into typed sub-column data.

[0106] The fifth step is to determine the values ​​of each field in each of the above-mentioned transformation sub-columns as the values ​​of each field to be processed.

[0107] Step 6: For each of the above fields to be processed, perform the following steps: The first sub-step is to determine the field name corresponding to the above field value to be processed as the field name to be processed.

[0108] The second sub-step involves determining the corresponding data storage mode based on the aforementioned field names to be processed. This data storage mode characterizes the method of storing the data. The data storage mode can be "bucketed storage," "low-frequency storage," or "type failure storage."

[0109] In practice, firstly, in response to determining that the aforementioned field names to be processed meet the preset bucketing conditions, "bucketed storage" can be determined as the data storage mode. The aforementioned bucketing conditions can be that the aforementioned field names to be processed contain preset page characters. These preset page characters can be pre-defined strings. For example, the preset page characters can be "url".

[0110] Then, in response to determining that the aforementioned field name to be processed meets a preset low-frequency condition, "low-frequency storage" can be determined as the data storage mode. The aforementioned low-frequency condition can be that the aforementioned field name to be processed contains a preset time character. The preset time character can be a string used to represent time. For example, the preset time character can be "time".

[0111] Then, in response to determining that the aforementioned field name to be processed meets the preset type failure condition, the aforementioned execution entity can determine "type failure storage" as the data storage mode. The aforementioned type failure condition can be that the aforementioned field name to be processed does not meet the aforementioned bucketing condition and the aforementioned low-frequency condition.

[0112] The third sub-step involves, in response to determining that the aforementioned data storage mode meets the preset bucketing conditions, performing dictionary encoding on the aforementioned field values ​​to be processed, obtaining encoded values ​​as sparse data. The bucketing conditions can be that the aforementioned data storage mode is "bucketed storage." The encoded values ​​to be processed can be the field values ​​to be processed after dictionary encoding. In practice, the executing entity can use a dictionary encoding algorithm to perform dictionary encoding on the aforementioned field values ​​to be processed, obtaining encoded field values ​​as encoded values. The dictionary encoding algorithm can be any algorithm capable of encoding data. For example, the dictionary encoding algorithm can be the LZW algorithm (Lempel-Ziv-Welch).

[0113] The fourth sub-step involves, in response to determining that the aforementioned data storage mode satisfies a preset low-frequency path condition, storing the aforementioned field name and value to be processed in a low-frequency associative container to update the low-frequency associative container, and identifying the updated low-frequency associative container as sparse data. The aforementioned low-frequency path condition can be that the aforementioned data storage mode is "low-frequency storage". The initial low-frequency associative container can be an empty associative container (Map). The updated low-frequency associative container can be a Map associative container that stores the aforementioned field name and value to be processed.

[0114] The fifth sub-step involves, in response to determining that the aforementioned data storage mode meets a preset type conversion failure condition, storing the aforementioned field name and value to be processed in a type conversion failure container to update the type conversion failure container, and defining the updated type conversion failure container as sparse data. The aforementioned type conversion failure condition can be that the aforementioned data storage mode is "type failure storage". The initial type conversion failure container can be an empty Map associative container. The updated type conversion failure container can be an associative container (Map) that stores the aforementioned field name and value to be processed.

[0115] The seventh step is to combine the identified sparse data into sparse column data.

[0116] The above technical solution and its related content, combined with step 109, serve as an inventive point of this disclosure, solving the problem of "large storage space occupied during storage." Factors leading to low efficiency often include: Variant cannot recognize the specific structure and content characteristics of semi-structured data, resulting in the inability to perform differentiated compression on different types of data during storage, leading to a large storage space occupied during storage. Solving these factors can reduce the storage space occupied during storage. To achieve this effect, this disclosure first combines the aforementioned unmatched sub-columns into dynamic sub-column area data. Second, it determines the field values ​​included in the aforementioned target transformation sub-columns as respective target field values. Then, for each of the aforementioned target field values, the following steps are performed: First, in response to determining that the aforementioned target field value meets preset discrete data conditions, the aforementioned target field value undergoes numerical encoding processing to obtain target field encoded data as target field feature information. Therefore, when the target field value is string data, numerical encoding processing can be performed on the target field value to compress it. Secondly, in response to determining that the target field value meets the preset continuous data condition, the target field value is identified as target field feature information. Therefore, when the target field value is numerical, it can be normalized to compress it. Then, in response to determining that the target field value meets the preset time data condition, it undergoes time feature transformation to obtain target time data as target field feature information. Therefore, when the target field value is data representing time, it can be converted to a timestamp to compress it. Then, the field name corresponding to the target field value is identified as the target field name. Next, the target field name and the obtained target field feature information are combined into typed key-value pairs. Then, the obtained typed key-value pairs are combined into typed sub-column data. Then, the field values ​​included in each of the above-mentioned sub-columns to be processed are identified as each field value to be processed. Then, for each of the above-mentioned field values ​​to be processed, the following steps are performed: Then, the field name corresponding to the above-mentioned field value to be processed is identified as the field name to be processed. Then, based on the aforementioned field names to be processed, a data storage mode corresponding to those field names is determined. Then, in response to determining that the aforementioned data storage mode meets preset bucketing conditions, the values ​​of the fields to be processed are dictionary-encoded to obtain encoded values ​​as sparse data. Thus, dictionary encoding can be performed on the values ​​of the fields to be processed, thereby compressing them. Next, in response to determining that the aforementioned data storage mode meets preset low-frequency path conditions, the aforementioned field names and values ​​to be processed are stored in a low-frequency association container to update the low-frequency association container and to determine the updated low-frequency association container as sparse data.Therefore, the field names and values ​​to be processed can be stored in a Map associative container. Then, in response to determining that the above data storage mode meets the preset type conversion failure condition, the field names and values ​​to be processed are stored in a type conversion failure container to update the type conversion failure container and determine the updated type conversion failure container as sparse data. Finally, the determined sparse data are combined into sparse column area data. Because different types of data can be compressed differently before storage, the data size can be effectively reduced, thus reducing the storage space occupied during storage.

[0117] Step 109: Perform columnar partitioning storage on the dynamic sub-column data, sparse column data, and typed sub-column data.

[0118] In some embodiments, the execution entity may perform columnar partitioning storage of the dynamic sub-column data, the sparse column data, and the typed sub-column data.

[0119] In addressing the technical problems mentioned above by adopting technical solutions, and considering the application scenario of storing semi-structured data in mines, the following technical issues often arise: when using Variant to store semi-structured data, only row-based storage is possible, which easily leads to low storage space utilization and wasted storage resources. Given the following requirements of this application scenario: the mine is located in a remote area with limited storage resources, and there is a need to store semi-structured data collected by various sensors in real time for real-time monitoring of the mine's environmental safety, it is necessary to improve storage space utilization and minimize storage resource waste to ensure the integrity of core safety data. Therefore, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the aforementioned execution entity may perform columnar partitioning storage of the aforementioned dynamic sub-column area data, the aforementioned sparse column area data, and the aforementioned typed sub-column area data through the following steps: The first step is to determine the preset initial storage format as the dynamic storage format corresponding to the aforementioned dynamic sub-column area data. This initial storage format can be a storage method pre-determined by technical personnel for the dynamic sub-column area data. For example, the initial storage format can be a file.

[0120] The second step involves storing each unmatched sub-column included in the dynamic sub-column area data to a dynamic disk partition, based on the aforementioned dynamic storage format. This dynamic disk partition can be a disk pre-defined by an engineer for storing the dynamic sub-column area data. In practice, in response to determining that the dynamic storage format is a file, the executing entity can store each unmatched sub-column included in the dynamic sub-column area data to the dynamic disk partition in the form of a file.

[0121] Third, for each sparse data point included in the sparse column region data above, in response to determining that the sparse data satisfies the preset container conditions, the following steps are performed: The first sub-step involves generating memory usage data and the number of sparse fields corresponding to the sparse data. Here, the container condition can be that the sparse data is an associative container, a Map. The memory usage data can be the number of bytes occupied by the sparse data in memory. The number of sparse fields can be the number of key-value pairs in the sparse data.

[0122] In practice, the aforementioned execution entity can serialize sparse data using a serialization interface to obtain a byte stream corresponding to the sparse data as a sparse byte stream. This serialization interface can be any interface capable of serializing data to generate a corresponding byte stream. For example, the serialization interface could be the Serializable interface. Then, the number of bytes in the sparse byte stream can be determined as the memory usage. Finally, the number of sparse fields in the sparse data can be obtained using a data verification function. This data verification function can be any function capable of obtaining the number of key-value pairs in a Map. For example, the data verification function could be the size() method in the Map interface.

[0123] The second sub-step involves generating target container memory data based on the aforementioned memory usage data and the number of sparse fields. This target container memory data can be the number of bytes that the sparse data should occupy in memory.

[0124] In practice, firstly, the memory usage data and a preset sparse value can be input into a maximum value function, and the return value of the maximum value function is used as the number of bytes occupied in the block. The maximum value function can be the `max()` function. The sparse value can be a preset value, for example, 1. Then, the ratio of the preset base block size to the above block size is determined as the data block factor. The base block size can be a preset value representing the number of bytes, for example, 1024. Next, the data block factor and the sparse value can be input into a minimum value function, and the return value of the minimum value function is used as the first factor. The minimum value function can be the `min()` function. Then, the sparse value and the number of sparse fields can be input into the maximum value function, and the return value of the maximum value function is used as the key-value pair quantity data. Finally, the ratio of a preset reference value to the key-value pair quantity data is determined as the key-value pair factor. The preset reference value can be a preset value, for example, 100. Then, the key-value pair factor and the sparse numerical value can be input into the minimum function to obtain the return value of the minimum function as the second factor. Then, the product of the first factor, the second factor, and the number of bytes in the base block can be determined as the target container memory data.

[0125] The third sub-step involves adaptively adjusting the sparse data based on the aforementioned memory usage data and the target container memory data to update the sparse data. In practice, in response to the determination that the aforementioned memory usage data is greater than the target container memory data, the sparse data is compressed using a compression method to update the sparse data. This compression method can be any method capable of compressing Map data. For example, the compression method could be from the FastUtil library or the Snappy library.

[0126] The fourth sub-step involves storing the updated sparse data to a sparse disk partition, thereby updating the sparse disk partition. The initial sparse disk partition can be a disk used to store sparse data. In practice, the executing entity can store the updated sparse data to this sparse disk partition.

[0127] The fourth step is to identify the sparse data that does not meet the above container conditions from the sparse data included in the above sparse column area as the sparse data to be stored.

[0128] The fifth step is to store all the sparse data to be stored in the updated sparse disk partition.

[0129] The sixth step involves grouping the typed key-value pairs included in the aforementioned typed sub-column data to obtain various typed key-value pair data groups. In practice, the executing entity can group typed key-value pairs with the same target field name into a single group to obtain various typed key-value pair data groups.

[0130] Step 7: For each typed key-value pair data group in the above-mentioned typed key-value pair data groups, perform the following steps: The first sub-step involves determining the target field names included in the aforementioned typed key-value pair data group as the group name.

[0131] The second sub-step, in response to determining that each typed key-value pair in the aforementioned typed key-value pair data group satisfies a preset feature condition, generates a data index structure corresponding to the aforementioned typed key-value pair data. The data index structure can be the name of the index to be built for the aforementioned typed key-value pair data group. For example, the data index structure can be an HNSW index (Hierarchical Navigable Small World) or a Zone Map index. The aforementioned feature condition can be that each typed key-value pair in the aforementioned typed key-value pair data group is a feature vector. In practice, the executing entity can determine the preset feature index structure as the data index structure. The feature index structure can be an HNSW index.

[0132] The third sub-step involves generating a data index structure corresponding to the aforementioned typed key-value pairs in response to determining that each typed key-value pair in the aforementioned typed key-value pair data group meets a preset numerical condition. The aforementioned numerical condition can be that the characteristic information of each target field included in each typed key-value pair data is numerical.

[0133] In practice, in response to determining that each typed key-value pair in the aforementioned typed key-value pair data group satisfies the above numerical conditions, a preset typed numerical index structure can be defined as the data index structure. This typed numerical index structure can be a Zone Map index.

[0134] The fourth sub-step involves constructing type columns based on the aforementioned typed key-value pair data groups, the aforementioned group names, and the generated data index structure.

[0135] The aforementioned type columns can be columns in a pre-defined type database used to store the aforementioned typed key-value pair data groups. Each type column includes a field name and its values. The field names can be the names of the type columns, i.e., the group names. The field values ​​can be the characteristic information of each target field included in the typed key-value pair data group. The pre-defined type database can be a pre-built Apache Doris database or a PostgreSQL database.

[0136] In practice, firstly, in response to determining that the above data index structure is a Zone Map index, the following steps are performed: the Apache Doris database can be identified as a columnar database. Then, a table can be created directly in the columnar database using SQL commands, and a column named after the aforementioned group name can be created in the created table as the initial type column. As an example, when the table created in the columnar database is named "sheet" and the group name is "user_id", a column can be created as the initial type column using the following command statement: "ALTER TABLE sheet ADD COLUMN user_id VARCHAR(65533)".

[0137] Secondly, in response to determining that the above data index structure is an HNSW index, the following steps are performed: First, the pgvector extension can be installed on the PostgreSQL database using command statements, and the installed PostgreSQL database can be selected as the database to be stored. For example, the pgvector extension can be installed on the PostgreSQL database using the statement "CREATE EXTENSION vector". Second, a table can be created in the database to be stored using SQL commands, and a column named after the above group name can be created in the created table as the column to be adjusted. Then, an HNSW index can be created on the column to be adjusted to update the column, and the updated column to be adjusted can be used as the initial type column.

[0138] Then, the characteristic information of each target field included in the above-mentioned typed key-value pair data group can be stored in the initial type column as the field values ​​of the initial type column to update the initial type column, and the updated initial type column is determined as the type column.

[0139] Step 8: Store the constructed type columns to a typed disk partition. This typed disk partition can be a disk used to store the updated type columns. In practice, the execution entity can store the updated type columns to the typed disk partition using columnar storage.

[0140] The above technical solution and its related content, combined with step 109, serve as an inventive point of this disclosure, solving the problem of "wasteful storage resources." Factors leading to low efficiency often include: when using Variant to store semi-structured data, only row-based storage of the semi-structured data is possible, which easily leads to low utilization of storage space and waste of storage resources. Solving these factors can reduce storage resource waste. To achieve this effect, this disclosure first determines the preset initial storage format as the dynamic storage format corresponding to the aforementioned dynamic sub-column area data. This allows the determination of the storage format for the dynamic sub-column area data. Second, based on the aforementioned dynamic storage format, each unmatched sub-column included in the aforementioned dynamic sub-column area data is stored to a dynamic disk partition. This allows each unmatched sub-column included in the dynamic sub-column area data to be stored to a dynamic disk partition. Then, for each sparse data in the aforementioned sparse column area data, in response to determining that the sparse data meets preset container conditions, the following steps are performed: First, based on the sparse data, memory usage data and the number of sparse fields corresponding to the sparse data are generated. Therefore, memory usage data and the number of sparse fields can be generated for sparse data. Next, based on the aforementioned memory usage data and the number of sparse fields, target container memory data is generated. This yields the target container memory data for the sparse data. Then, based on the aforementioned memory usage data and the target container memory data, the sparse data is adaptively adjusted to update it. This compresses the size of the sparse data. The updated sparse data is then stored in a sparse disk partition to update the sparse disk partition. Next, the sparse data in the sparse column area data that does not meet the aforementioned container conditions is identified as individual sparse data to be stored. These individual sparse data to be stored are then stored in the updated sparse disk partition. Finally, the typed key-value pairs in the typed sub-column area data are grouped to obtain individual typed key-value pair data groups. Then, for each of the aforementioned typed key-value pair data groups, the following steps are performed: First, the target field names included in the aforementioned typed key-value pair data group are determined as the group name. Second, in response to determining that each typed key-value pair data included in the aforementioned typed key-value pair data group satisfies a preset characteristic condition, a data index structure corresponding to the aforementioned typed key-value pair data is generated. Thus, the index structure of the typed key-value pair data group can be determined. Then, in response to determining that each typed key-value pair data included in the aforementioned typed key-value pair data group satisfies a preset numerical condition, a data index structure corresponding to the aforementioned typed key-value pair data is generated. Thus, the index structure of the typed key-value pair data group can be determined.Then, based on the aforementioned typed key-value pair data groups, the aforementioned group names, and the generated data index structure, typed columns are constructed. This allows for the construction of typed columns in a columnar database. Finally, each constructed typed column is stored to a typed disk partition. This allows for columnar storage of each typed column to a typed disk partition. Because the data is compressed before storage if it is large, the storage space occupied during data storage can be reduced. Furthermore, because data can be stored on disk in a columnar manner, storage space utilization can be improved, and storage resource waste can be reduced.

[0141] The above embodiments of this disclosure have the following beneficial effects: the columnar partitioning storage method for variant types in some embodiments of this disclosure can reduce storage resource waste. Specifically, the reason for storage resource waste is that directly using the Variant data type to store semi-structured data easily leads to the storage of a large amount of redundant data such as duplicate key names or punctuation marks, resulting in storage resource waste. Based on this, the columnar partitioning storage method for variant types in some embodiments of this disclosure firstly, in response to receiving initial semi-structured data and type definition syntax sent by the user client, performs syntax embedding processing on the variant type based on the type definition syntax to obtain the pattern template statement corresponding to the variant type. Thus, the variant type can be defined. Secondly, the pattern template statement is subjected to data extraction processing to obtain each template key-value pair data. Thus, each template key-value pair data can be obtained. Then, based on the above template key-value pair data, pattern template metadata is generated. Thus, pattern template metadata can be obtained. Then, the initial semi-structured data is subjected to data parsing processing to obtain each initial key-value pair data. Thus, the initial semi-structured data can be parsed to obtain the key-value pairs in the initial semi-structured data. Next, based on the aforementioned schema template metadata, hierarchical data matching is performed on each initial key-value pair, resulting in composite semi-structured data and each unmatched key-value pair. The composite semi-structured data includes each composite key-value pair. This determines whether each initial key-value pair matches the schema template metadata and filters out the mismatched initial key-value pairs. Then, transformation and verification processing is performed on the unmatched key-value pairs and the composite key-value pairs included in the composite semi-structured data, resulting in composite transformed sub-columns and unmatched sub-columns. This allows the unmatched key-value pairs and composite semi-structured data to be converted into columns. Then, data filtering processing is performed on the composite transformed sub-columns, resulting in target transformed sub-columns and unprocessed transformed sub-columns. This allows for further filtering of the composite transformed sub-columns, resulting in target transformed sub-columns and unprocessed transformed sub-columns. Next, the unmatched sub-columns, target transformed sub-columns, and unprocessed transformed sub-columns are partitioned and encoded to obtain dynamic sub-column data, sparse column data, and typed sub-column data. This allows for the determination of different storage partitions for different data. Finally, the dynamic sub-column data, sparse column data, and typed sub-column data are stored using columnar partitioning. This allows for partitioned storage of different data. Furthermore, because the semi-structured data can be converted into key-value pairs before storage, and then these key-value pairs are converted into columns, a large number of punctuation marks and duplicate key names can be removed from the semi-structured data, reducing storage resource waste.

[0142] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a columnar partitioned storage device for variant types, these device embodiments being similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0143] like Figure 2 As shown, some embodiments of the columnar partitioned storage device 200 for variant types include: a syntax embedding unit 201, a data extraction unit 202, a generation unit 203, a data parsing unit 204, a hierarchical matching unit 205, a conversion verification unit 206, a data filtering unit 207, a partition encoding unit 208, and a storage unit 209. The system comprises the following components: a syntax embedding unit 201, configured to, in response to receiving initial semi-structured data and type definition syntax sent by a user client, perform syntax embedding processing on variant types based on the type definition syntax to obtain pattern template statements corresponding to the variant types; a data extraction unit 202, configured to perform data extraction processing on the pattern template statements to obtain each template key-value pair data; a generation unit 203, configured to generate pattern template metadata based on the template key-value pair data; a data parsing unit 204, configured to perform data parsing processing on the initial semi-structured data to obtain each initial key-value pair data; and a hierarchical matching unit 205, configured to perform hierarchical matching processing on the initial key-value pair data based on the pattern template metadata to obtain composite semi-structured data and each unmatched key-value pair data. The aforementioned composite semi-structured data includes various composite key-value pairs; the transformation verification unit 206 is configured to perform transformation verification processing on the aforementioned unmatched key-value pairs and the composite key-value pairs included in the aforementioned composite semi-structured data, to obtain various composite transformation sub-columns and various unmatched sub-columns; the data filtering unit 207 is configured to perform data filtering processing on the aforementioned composite transformation sub-columns, to obtain various target transformation sub-columns and various transformation sub-columns to be processed; the partitioning encoding unit 208 is configured to perform partitioning encoding processing on the aforementioned unmatched sub-columns, the aforementioned target transformation sub-columns, and the aforementioned transformation sub-columns to be processed, to obtain dynamic sub-column area data, sparse column area data, and typed sub-column area data; the storage unit 209 is configured to perform columnar partitioning storage on the aforementioned dynamic sub-column area data, the aforementioned sparse column area data, and the aforementioned typed sub-column area data.

[0144] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0145] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (such as a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0146] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0147] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0148] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0149] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0150] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0151] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: respond to receiving initial semi-structured data and type definition syntax sent by a user client; perform syntax embedding processing on variant types based on the aforementioned type definition syntax to obtain pattern template statements corresponding to the aforementioned variant types; perform data extraction processing on the aforementioned pattern template statements to obtain each template key-value pair data; generate pattern template metadata based on the aforementioned template key-value pair data; perform data parsing processing on the aforementioned initial semi-structured data to obtain each initial key-value pair data; and perform data hierarchical matching processing on the aforementioned initial key-value pair data based on the aforementioned pattern template metadata to obtain a composite semi-structured... The process involves transforming the data and each unmatched key-value pair, wherein the aforementioned composite semi-structured data includes each composite key-value pair; performing transformation and verification processing on the aforementioned unmatched key-value pair data and the composite key-value pair data included in the aforementioned composite semi-structured data to obtain each composite transformed sub-column and each unmatched sub-column; performing data filtering processing on the aforementioned composite transformed sub-column to obtain each target transformed sub-column and each transformed sub-column to be processed; performing partitioning and encoding processing on the aforementioned unmatched sub-column, the aforementioned target transformed sub-column, and the aforementioned transformed sub-column to be processed to obtain dynamic sub-column area data, sparse column area data, and typed sub-column area data; and performing columnar partitioning storage on the aforementioned dynamic sub-column area data, the aforementioned sparse column area data, and the aforementioned typed sub-column area data.

[0152] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0154] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a syntax embedding unit, a data extraction unit, a generation unit, a data parsing unit, a hierarchical matching unit, a conversion verification unit, a data filtering unit, a partitioning encoding unit, and a storage unit. The names of these units do not necessarily limit the unit itself; for example, the storage unit may also be described as "a unit that performs columnar partitioning storage of the aforementioned dynamic sub-column area data, the aforementioned sparse column area data, and the aforementioned typed sub-column area data."

[0155] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0156] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for variant type columnar partition storage, comprising: in response to receiving initial semi-structured data and type definition syntax sent by a user client, performing syntax embedding processing on a variant type based on the type definition syntax to obtain a schema template statement corresponding to the variant type; performing data extraction processing on the schema template statement to obtain each template key-value pair data; generating schema template metadata based on the each template key-value pair data; performing data parsing processing on the initial semi-structured data to obtain each initial key-value pair data; performing data hierarchical matching processing on the each initial key-value pair data based on the schema template metadata to obtain composite semi-structured data and each unmatched key-value pair data, wherein the composite semi-structured data comprises each composite key-value pair data; performing conversion verification processing on the each unmatched key-value pair data and each composite key-value pair data included in the composite semi-structured data to obtain each composite conversion sub-column and each unmatched sub-column; performing data screening processing on the each composite conversion sub-column to obtain each target conversion sub-column and each to-be-processed conversion sub-column; performing partition encoding processing on the each unmatched sub-column, the each target conversion sub-column, and the each to-be-processed conversion sub-column to obtain dynamic sub-column region data, sparse column region data, and typed sub-column region data; performing columnar partition storage on the dynamic sub-column region data, the sparse column region data, and the typed sub-column region data.

2. The method of claim 1, wherein, The data screening processing on the each composite conversion sub-column to obtain each target conversion sub-column and each to-be-processed conversion sub-column comprises: for each composite conversion sub-column in the each composite conversion sub-column, the following steps are performed: in response to determining that the composite conversion sub-column satisfies a preset sub-column condition, determining the composite conversion sub-column as a target conversion sub-column; in response to determining that the composite conversion sub-column does not satisfy the sub-column condition, determining the composite conversion sub-column as a to-be-processed conversion sub-column.

3. The method of claim 1, wherein, Each template key-value pair data in the each template key-value pair data comprises a path schema, a target type, and an annotation description; and the generating schema template metadata based on the each template key-value pair data comprises: for each template key-value pair data in the each template key-value pair data, the following steps are performed: performing character segmentation processing on a path schema included in the template key-value pair data based on a preset delimiter to obtain each path string; for each path string in the each path string, in response to determining that the path string satisfies a preset string condition, performing string verification processing on the path string to obtain a string verification result; in response to determining that each obtained string verification result satisfies a preset verification condition, determining each path string in the each path string that does not satisfy the string condition as each to-be-processed path string; performing wildcard verification on the each to-be-processed path string to obtain each wildcard verification result; In response to determining that the wildcard verification results meet preset wildcard verification conditions, performing type detection processing on the target types included in the template key-value pair data to obtain type detection results; In response to determining that the obtained type detection results meet preset type detection conditions, performing conflict detection processing on the path patterns included in the template key-value pair data to obtain conflict detection results; In response to determining that the conflict detection results meet preset conflict detection conditions, performing integration processing on the template key-value pair data to obtain pattern template metadata.

4. The method of claim 3, wherein, The conflict detection processing on the path patterns included in the template key-value pair data to obtain conflict detection results includes: For each path pattern in the path patterns included in the template key-value pair data, the following steps are performed: The path pattern is determined as a to-be-matched path pattern; Path patterns in the path patterns that meet preset pattern conditions are determined as to-be-processed path patterns; For each to-be-processed path pattern in the to-be-processed path patterns, data comparison processing is performed on the to-be-processed path pattern and the to-be-matched path pattern to obtain a path comparison result; In response to determining that the obtained path comparison results meet preset conflict conditions, preset conflict information is determined as a conflict detection result.

5. The method of claim 4, wherein, The method further includes: In response to determining that the obtained path comparison results do not meet the conflict conditions, the following steps are performed: For each path pattern in the path patterns included in the template key-value pair data, character segmentation processing is performed on the path pattern to obtain a path segmentation character string sequence; Path segmentation characters in the obtained path segmentation character string sequences that meet preset data conditions are determined as first-level segmentation characters; For each first-level segmentation character in the first-level segmentation characters, the following steps are performed: The first-level segmentation character is determined as a target first-level character; First-level segmentation characters in the first-level segmentation characters that meet preset data matching conditions are determined as to-be-processed first-level characters; For each to-be-processed first-level character in the to-be-processed first-level characters, data comparison is performed on the to-be-processed first-level character and the target first-level character to obtain a first-level comparison result; In response to determining that the obtained first-level comparison results meet preset first-level comparison conditions, preset non-conflict information is determined as a conflict detection result; In response to determining that the obtained first-level comparison results do not meet the first-level comparison conditions, first-level comparison results in the first-level comparison results that meet preset first-level result conditions are determined as target first-level comparison results; Path patterns corresponding to the target first-level comparison results are determined as first-level path patterns; Target types corresponding to the first-level path patterns are determined as first-level target types; In response to determining that the first-level target types do not meet preset type same conditions, the conflict information is determined as a conflict detection result.

6. The method of claim 1, wherein, The schema template metadata includes a plurality of field groups, each of the plurality of field groups includes a path field and a type field, the path field corresponds to a path schema, and the type field corresponds to a target type, and each of the plurality of initial key-value pair data includes a field name and a field value; and the data hierarchical matching unit is configured to perform data hierarchical matching processing on the plurality of initial key-value pair data based on the schema template metadata, to obtain composite semi-structured data and a plurality of unmatched key-value pair data, including: For each of the plurality of initial key-value pair data, the following steps are performed: determining the field name and the field value included in the initial key-value pair data as an initial field name and an initial field value, respectively; determining each path field in each of the plurality of field groups included in the schema template metadata as a template path field; determining each path schema corresponding to each template path field as a template path field value; finding a template path field value corresponding to the initial field name from the plurality of template path field values to obtain a field value finding result; in response to determining that the field value finding result does not satisfy a preset finding condition, determining the initial key-value pair data as an unmatched key-value pair data; in response to determining that the field value finding result satisfies the finding condition, determining a template path field value corresponding to the initial field name in the plurality of template path field values as a target template path field value; determining a target type of a type field corresponding to the target template path field value as a target type field value; performing data conversion processing on the initial field value based on the target type field value to obtain a converted field value; combining the initial field name, the target type field value, and the converted field value as a composite key-value pair data; generating composite semi-structured data based on the plurality of obtained composite key-value pair data.

7. A columnar partition storage device for variant types, comprising: a syntax embedding unit configured to, in response to receiving initial semi-structured data and type definition syntax sent by a user client, perform syntax embedding processing on a variant type based on the type definition syntax to obtain a schema template statement corresponding to the variant type; a data extraction unit configured to perform data extraction processing on the schema template statement to obtain a plurality of template key-value pair data; a generation unit configured to generate schema template metadata based on the plurality of template key-value pair data; a data parsing unit configured to perform data parsing processing on the initial semi-structured data to obtain a plurality of initial key-value pair data; a hierarchical matching unit configured to perform data hierarchical matching processing on the plurality of initial key-value pair data based on the schema template metadata to obtain composite semi-structured data and a plurality of unmatched key-value pair data, wherein the composite semi-structured data includes a plurality of composite key-value pair data; a conversion verification unit configured to perform conversion verification processing on the plurality of unmatched key-value pair data and the plurality of composite key-value pair data included in the composite semi-structured data to obtain a plurality of composite conversion sub-columns and a plurality of unmatched sub-columns; The data screening unit is configured to perform data screening processing on the respective composite conversion sub-columns to obtain respective target conversion sub-columns and respective to-be-processed conversion sub-columns; The partition encoding unit is configured to perform partition encoding processing on the respective unmatched sub-columns, the respective target conversion sub-columns and the respective to-be-processed conversion sub-columns to obtain dynamic sub-column region data, sparse column region data and typed sub-column region data; The storage unit is configured to perform columnar partition storage on the dynamic sub-column region data, the sparse column region data and the typed sub-column region data. 8.An electronic device, comprising: one or more processors; one or more programs stored on the storage device; when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having stored thereon a computer program, wherein, The program is executed by the processor to implement the method according to any one of claims 1 to 6.

Images