Efficient storage and querying of schema-less data

Abstract

A method (300) of storing semi-structured data (12U) includes receiving user data (12) comprising semi-structured user data from a user (10) of a query system (150). The method includes receiving an indication (14) that the semi-structured user data fails to include a fixed schema. In response, the method further includes parsing the semi-structured user data into a plurality of data paths (210) and extracting a data type (220) associated with each respective data path of the plurality of data paths. The method additionally includes storing the semi-structured user data as a row entry in a table (204) of a database in communication with the query system, wherein each column value associated with the row entry corresponds to a respective one of the plurality of data paths and the data type associated with the respective data path.

Description

Technical Field

[0001] This disclosure relates to efficient storage and retrieval of schema-less data. Background Technology

[0002] As today's applications generate massive amounts of data, query systems and other analytical tools continue to evolve to support data analysis. Furthermore, the value of generating large amounts of user data can be significantly diminished if users cannot perform analysis on their data in an efficient and / or cost-effective manner. To ensure users can analyze their data, query processing systems have begun to operate in conjunction with data storage systems. Through collaboration, user data can be stored in storage structures that facilitate analytical operations such as queries or other analyses. Unfortunately, the data storage structures that facilitate these operations are often limited by their ability to support semi-structured or schema-less data. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method for storing semi-structured data. When executed by data processing hardware, the method causes the data processing hardware to perform operations. The operations include receiving user data from a user of a query system, wherein the user data includes unstructured user data. The operations also include receiving an indication that the unstructured user data fails to include a fixed pattern. In response to the indication that the unstructured user data fails to include a fixed pattern, the operations further include parsing the unstructured user data into multiple data paths and extracting a data type associated with each corresponding data path among the multiple data paths. The operations further include storing the unstructured user data as row entries in a table of a database communicating with the query system, wherein each column value associated with a row entry corresponds to a corresponding one of the multiple data paths and a data type associated with that corresponding data path.

[0004] Another aspect of this disclosure provides a system capable of storing unstructured data. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving user data from a user of a query system, wherein the user data includes unstructured user data. The operations also include receiving an indication that the unstructured user data fails to include a fixed pattern. In response to the indication that the unstructured user data fails to include a fixed pattern, the operations further include parsing the unstructured user data into multiple data paths and extracting a data type associated with each corresponding data path among the multiple data paths. The operations further include storing the unstructured user data as row entries in a table of a database communicating with the query system, wherein each column value associated with a row entry corresponds to a corresponding one of the multiple data paths and a data type associated with that corresponding data path.

[0005] Implementations of the methods or systems disclosed herein may include one or more of the following optional features. In some implementations, the user data further includes structured user data having a corresponding fixed pattern, and the tables of the database include one or more row entries corresponding to the structured user data. In some examples, the unstructured user data includes JavaScript Object Notation (JSON). In these examples, the corresponding column associated with the row entry may include an explicit null value. Also in these examples, the corresponding column value associated with the corresponding column associated with the row entry may include an empty array. In some configurations, storing unstructured user data as row entries in a table of the database further includes identifying a first data type and a second data type associated with a first data path among a plurality of data paths parsed from the unstructured data. In these configurations, row entries storing unstructured user data as row entries in a table of the database further include storing a first value of the unstructured user data corresponding to the first data type of the first data path in a first row entry of the table, and storing a second value of the unstructured user data corresponding to the second data type of the first data path in a second row entry of the table in a first column. Here, the second data type is a data type different from the first data type.

[0006] In some examples, the operations for a method or system further include: receiving, during query runtime, a query from a user of the query system for data associated with stored unstructured data; determining the appropriate data path of the stored unstructured data in response to the query; and generating a query response that includes corresponding column values ​​for row entries corresponding to the appropriate data path of the stored unstructured data in response to the query. User data may correspond to log data of cloud computing resources associated with the user. Each of the multiple data paths may correspond to a key of a key-value pair of unstructured user data. The corresponding column values ​​for row entries may include nested arrays.

[0007] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of an example data management environment.

[0009] Figure 2A-2H It is used for Figure 1 A schematic diagram of an example schema manager for a data management environment.

[0010] Figure 3 This is a flowchart illustrating an example layout of operations for storing unstructured data.

[0011] Figure 4 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0012] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0013] Data storage systems can store user or client data in one or more large, queryable tables. The general structure of a table consists of data organized into rows called row entries, representing individual records. The length of a row can vary based on the table's schema and / or the number of columns or fields associated with a particular record (i.e., a row). A table schema is a specified format for a table that defines column names (e.g., fields), the data types of specific columns, and / or other information. In other words, the table schema is an integrity constraint imposed on the table, instructing how the table is organized. For example, a table schema might constrain the type or form of data in specific columns of one or more row entries. As an example, the schema for a specific column of a table might be a name field as a string, and the values ​​of row entries for that column would be names with string data types.

[0014] In some implementations, the storage system is configured to generate a table schema based on the attributes of the user data it receives. For example, the storage system receives user data in a row-oriented format with a specific schema before ingestion. In other implementations, the user or client coordinates with the storage system to define a schema for the user data before any transfer of the user data to the storage system. In any of these cases, the ability to query the stored user data depends on how the user data is stored within the data storage system. Therefore, the table schema influences query processing when the data used for records is stored as row entries in a table based on the table schema. For this purpose, structured data (which refers to data that includes a specified schema) can facilitate ingestion and enable efficient read and / or write operations on the ingested data during query processing. For example, each row of the table corresponds to a single purchase order from a parts supplier, and the columns of the table include a schema such as the name of the parts supplier (i.e., the manufacturer's name), the parts purchased, the cost of the parts, the address of the supplier (i.e., the manufacturer), and the supplier identifier (i.e., the company identifier). When the storage system receives a new purchase order record, the storage system organizes the data within the record into the appropriate columns of the table. Using this type of organization, the storage system can respond to queries from a query system by planning one or more actions (e.g., read or write operations) using a table schema. To illustrate, a client can use a query system that works in coordination with the storage system to inquire which purchase orders occurred within a specific date range. In response to this query, the query processing generates a query plan that sorts the purchase orders by date and then filters out all dates that do not correspond to the specific date range specified by the query. The query processing then returns the sorted and filtered results as a response to the query. In other words, here, the query plan executed by the query processing utilizes a schema of tables containing purchase orders to efficiently fulfill the query (e.g., with minimal latency).

[0015] Unfortunately, the reality is that user data exists in all forms. This means that while structured user data with a specified schema may be preferred for ease of ingestion and / or query processing, user data can also be unstructured or semi-structured. Here, unstructured or semi-structured data refers to data that does not have a fixed schema. For example, unstructured data can refer to data that is entirely or completely schema-less, while semi-structured data can refer to partially schema-less data. Therefore, the term unstructured or semi-structured data refers to data in which some parts of the data do not have a fixed schema. Such unstructured or semi-structured data can also be called “schemaless” data because the structure of the data can change from entry to entry (i.e., record to record). Therefore, from a storage perspective, how to organize a table of multiple records derived from schemaless data is not always obvious. Due to the dynamic and non-fixed nature of at least partially unstructured data, this unstructured data has traditionally presented problems for query systems and underlying storage systems. For example, without a fixed schema, the ingestion process may struggle to columnarize at least partially unstructured data into a table in a way that is efficient for query processing.

[0016] To address the problem of data synthesis, clients or users may not want to specify a particular schema for portions of unstructured data. For example, a client may lack confidence in their ability to generate a meaningful schema from a query processing and / or data storage perspective. Additionally or alternatively, user data may correspond to data automatically generated for the user (e.g., machine-generated data). As an example, a user might use a cloud computing environment that reports the status of the user's data (or resources) by generating log files. In this case, having the user generate a schema for this form of user data (e.g., user data log files) might be burdensome or beyond their capabilities. In other words, the user may not know what relevant schema should be specified for these types of files. In these scenarios, query and / or storage systems therefore require user-independent methods.

[0017] Furthermore, unstructured or semi-structured data is becoming increasingly popular because certain unstructured data formats are easy to create. This is especially true in machine-generated contexts, when data can be easily generated automatically in parts of an unstructured form (e.g., in automatically generated log files). One such popular data format is JavaScript Object Notation (JSON). JSON data is an increasingly common schema-less file format commonly used as a data exchange format. JSON is popular, in part because it is language-agnostic; meaning JSON is compatible with all programming languages. Additionally, JSON is a text-based and lightweight format, which allows it to be easily generated, whether by a user or a machine. Although JSON data has some form of text structure and key-value or attribute-value representation, its structure is not fixed from one record to another. Because JSON data lacks a consistent structure, querying JSON data efficiently is difficult, partly because query processing cannot use a formal schema in query planning or query validation.

[0018] Specifically, for JSON, several methods already exist for performing query processing on JSON data. In one approach, some systems interpret the entire JSON file as a string. While this preserves the data in the JSON file, it introduces problems for query systems because much of the efficiency of query systems (e.g., query speed) revolves around a fixed structure or pattern. In other words, the fixed pattern associated with data entries allows query systems to develop efficient query plans. Some of the efficiency in query planning and execution is that it is not necessary to read one or more data entries during query processing. However, when a storage system stores a JSON file as a string, queries targeting a specific entry or record in the JSON file often force query processing to address the entire string rather than just a portion of the entry. That is, query processing typically has to read the entire string to identify the relevant portion used for the response to the query. In this sense, string-based storage of JSON data has traditionally been undesirable for query systems.

[0019] To address some of the problems arising from data in which some parts are unstructured (e.g., unstructured or semi-structured data), the method described herein facilitates efficient storage and / or retrieval of such data (e.g., unstructured or semi-structured data) by extracting patterns from the data. While this method generally refers to semi-structured data below, the detailed approach can be applied to any data that is entirely (e.g., unstructured) or partially schema-less. By extracting patterns from semi-structured data, this method also avoids user dependence during schema generation for semi-structured data. Here, operations during ingestion process the semi-structured data and slice it into a columnar format to extract patterns from the semi-structured data for storage and querying purposes. Utilizing semi-structured data stored based on its extracted patterns in the storage system, when a query requests only a portion of the semi-structured data, the extracted patterns allow query processing to occur only at the requested portion to generate query results (also known as query responses). For example, when a query requests a portion of semi-structured data in a specific column of row entries, the query processing uses the extracted patterns to read only the values ​​in that specific column, rather than reading the entire semi-structured data (or a larger portion thereof).

[0020] The extraction method also offers the additional advantage that both semi-structured and structured data can be ingested by the storage system during the ingestion process. In other words, during ingestion, the storage system ingests a single block of user data containing both structured and semi-structured data for each row. Being able to handle both forms of data can be advantageous for users because they don't have to initiate two separate ingestion processes (i.e., one for structured data and one for semi-structured data). For example, records with structured data can have semi-structured fields, allowing the storage system to store the unstructured portion of the data in the same table as the structured data. In this respect, query results can return responses that include data from both structured and semi-structured data; thus allowing the query process to be more robust across different data structures.

[0021] Figure 1 An example of a data management environment 100 is shown. User device 110 associated with user 10 generates user data 12 during the execution of its computing resources 112 (e.g., data processing hardware 114 and / or memory hardware 116). For example, user 10 uses an application operating on the data processing hardware 114 of user device 110 to generate user data 12. Because various applications have the capability to generate user data 12, user 10 typically utilizes other systems (e.g., remote system 130, storage system 140, or query system 150) for user data storage and / or user data management.

[0022] In some examples, user equipment 110 is a local device (e.g., associated with the location of user 10) that uses its own computing resources 112 and has the ability to communicate (e.g., via network 120) with one or more remote systems 130. Additionally or alternatively, user equipment 110 utilizes its access to remote resources (e.g., remote computing resource 132) to operate applications for user 10. User data 12 generated using user equipment 110 may initially be stored locally (e.g., in data storage 118 such as in memory hardware 116) and then transferred to or sent to remote system 130 via network 120 at creation time. For example, user equipment 110 uses remote system 130 to transfer user data 12 to storage system 140.

[0023] In some examples, user 10 utilizes computing resources 132 of remote system 130 (e.g., a cloud computing environment) to store user data 12. In these examples, remote system 130 may receive user data 12 as it is generated by various user applications (e.g., streaming data). Here, a data stream (e.g., a stream of user data 12) refers to a continuous or substantially continuous data feed arriving at remote system 130 for storage and / or further processing. In some configurations, instead of continuously streaming user data 12 to remote system 130, user 10 and / or remote system 130 configure user data 12 to be sent in batches (e.g., at intervals) for processing. Much like user equipment 110, remote system 130 includes computing resources 132, such as remote data processing hardware 134 (e.g., servers and / or CPUs) and memory hardware 136 (e.g., disks, databases, or other forms of data storage).

[0024] In some configurations, remote computing resources 132 are resources utilized by various systems associated with and / or communicating with remote system 130. For example... Figure 1 As shown, these systems may include storage system 140 and / or query system 150. In some examples, the functionality of these systems 140, 150 may be arranged in different ways (e.g., built upon each other) or integrated together as separate systems capable of communicating with each other. For example, storage system 140 and query system 150 may be combined into a single system (e.g., as shown in the diagram). Figure 1 (These systems are shown by dashed lines in the diagram). A remote system 130 having its computing resources 132 can be configured to host one or more functions of these systems 140, 150. In some embodiments, the remote system 130 is a distributed system, with its computing resources 132 distributed across one or more locations accessible via network 120.

[0025] In some examples, storage system 140 is configured to operate data warehouse 142 (e.g., data storage and / or multiple databases) as a data storage device for user 10 (or multiple users). Generally, data warehouse 142 stores data from one or more sources and can be designed to analyze, report on, and / or integrate data from its sources. Data warehouse 142 enables users (e.g., organizational users) to have a central repository and storage data access point. By including user data 12 in a central repository such as data warehouse 142, data warehouse 142 can streamline data retrieval for functions such as data analysis and / or data reporting (e.g., by query system 150 and / or analysis system). Furthermore, data warehouse 142 can be configured to store large amounts of data, allowing user 10 (e.g., organizational users) to store large amounts of historical data to understand data trends. Since data warehouse 142 can be the primary or sole data repository for user data 12, storage system 140 can typically receive large amounts of data (e.g., gigabytes per second, terabytes per second, or more) from user devices 110 associated with user 10. Additionally or alternatively, as storage system 140, storage system 140 and / or storage warehouse 142 can be configured for multiple users from a single data source (e.g., multiple employees of an organization) and / or for data security (e.g., data redundancy) for simultaneous multi-user access. In some configurations, data warehouse 142 is persistent and / or non-volatile, such that default data is not overwritten or erased by newly incoming data.

[0026] Query system 150 is configured to request information or data from storage system 140 in the form of query 160. In some examples, query 160 is initiated by user 10 as a request for user data 12 within storage system 140 (e.g., an export data request). For example, user 10 interacts with query system 150 (e.g., an interface associated with query system 150) to retrieve user data 12 stored in data warehouse 142 of storage system 140. Here, query 160 can be user-initiated (i.e., requested directly by user 10) or system-initiated (i.e., configured by query system 150 itself). In some examples, query system 150 configures routines or recurring queries 160 (e.g., at a specified frequency) to allow user 10 to perform analysis or monitoring of user data 12 stored in storage system 140.

[0027] The format of query 160 can vary, but it typically includes a reference to specific user data 12 stored in storage system 140. Here, query system 150 operates in conjunction with storage system 140 such that storage system 140 stores user data 12 according to a specific schema, and query system 150 can generate query 160 based on information about the specific schema. For example, data storage system 140 stores user data 12 in a tabular format, where user data 12 populates the rows and columns of the table. Using the tabular format, user data 12 within the table has rows and columns corresponding to the schema associated with user data 12. For example, user data 12 could refer to a purchase order made by user 10. In this example, one or more tables storing user data 12 could include columns for the parts purchased, the cost of the parts, the manufacturer of the purchased parts, and details about the manufacturer (e.g., manufacturer address and / or manufacturer company identifier). Here, each row corresponds to a record or entry (e.g., a purchase order), where each column value associated with a row entry in the table's columns corresponds to a specific part of the schema. Since storage system 140 can receive user data 12 according to a specific pattern (e.g., a pattern indicated by pattern manager 200), storage system 140 is configured to store user data 12 such that query system 150 can access elements of the format (e.g., relation, header, or other pattern) associated with user data 12 (e.g., providing further context or definition to user data 12).

[0028] In response to query 160, query system 150 generates a query response 162 that fulfills or attempts to fulfill the request for query 160 (e.g., a request for specific user data 12). Generally, query response 162 includes the user data 12 requested by query system 150 in query 160. Query system 150 may return the query response 162 to the entity that initiated query 160 (e.g., user 10) or another entity or system communicating with query system 150. For example, query 160 itself or query system 150 may specify that query system 150 deliver one or more query responses 162 to a system associated with user 10, such as an analytics system. For example, user 10 uses the analytics system to perform analysis on user data 12. Typically, query system 150 is configured to generate routine queries 160 on user data 12 within storage system 140 to enable the analytics system to perform its analysis (e.g., at a specific frequency). For example, query system 150 executes daily query 160 to pull user log data from the analytics system for the last seven days for analysis and / or representation.

[0029] In some implementations, when query system 150 receives input for query 160, query system 150 is configured to determine query plan 152 to execute query 160. In other words, query 160 typically involves large tables at a basic level without specifically referencing the actual structure of the tables in storage system 140. For example, query 160 simply states that it queries the tables of user data 12 in storage system 140 to derive purchase data from manufacturers targeting Sprocket Labs over the past six months. To facilitate use as a user interface to extract more complex tables and / or storage structures from user data 12 in storage system 140, the input format for query 160 is simplified. Therefore, the user 10 executing or writing query 160 does not need to know the actual storage structure, but only the schema or fields of the high-level table structure to generate query 160. The query system 150, in conjunction with the storage system 140, can decompose the query 160 from the user 10 and rewrite the query 160 into a format that identifies potential operators (e.g., read / write operations) on the user data 12 to execute the query 160 on the underlying structure of the user data 12. That is, when the query system 150 receives the query 160, it digests the query 160 and plans how to execute the query 160 on the actual structure of the storage system 140. This planning may require identifying subsets of tables (e.g., partitions) and / or the files corresponding to the tables in the query 160.

[0030] refer to Figure 1 and Figure 2A-2HThe data management environment 100 also includes a schema manager 200 (also called manager 200). Manager 200 is configured to manage schema extraction during user data ingestion. Here, schema extraction refers to the process of generating a schema for semi-structured user data 12, 12U during ingestion and storage (e.g., by storage system 140). Schema extraction typically occurs during ingestion, but depending on the configuration requirements of a larger system, it can occur as a preprocessing step before ingestion. The semi-structured user data 12U, in contrast to the structured user data 12, 12S, does not include a fixed schema, so manager 200 performs schema extraction. In the absence of a fixed schema, manager 200 instead extracts a schema for the semi-structured user data 12U, which allows user data 12 (e.g., both structured data 12S and semi-structured data 12U) to be segmented and stored in storage system 140. When user data 12 is actually loaded into storage system 140 (e.g., during ingestion), schema extraction by manager 200 occurs, causing manager 200 to specify how user data 12 will be stored in one or more tables 204 of storage system 140. Manager 200 can extract the schema of semi-structured user data 12U by performing and / or coordinating operations related to systems 140, 150 of user 10 (e.g., storage operations and / or query operations). For example, manager 200 analyzes semi-structured user data 12U "on-the-fly" as it is processed and encoded during ingestion (e.g., becoming a low-level storage component). In this respect, manager 200 considers the characteristics of semi-structured data (e.g., data path and / or data type) when deciding how and where to store / encode the semi-structured data. That is, manager 200 does not necessarily build the schema of semi-structured data 12U as a preprocessing step before ingestion and then apply the schema, but rather interprets the schema of semi-structured data 12U as ingestion occurs. For example, this means that at any given moment, there may be no clearly defined pattern.

[0031] Depending on its design, the functionality of manager 200 can be centralized (e.g., residing in one of systems 140 and 150) or distributed among systems 140 and 150. In some examples, such as Figure 1Manager 200 is configured to receive user data 12 from user 10 and facilitate storage operations at storage system 140. For example, manager 200 facilitates a data loading request from user 10. In response to the loading request from user 10, manager 200 ingests user data 12 and may transform user data 12 into a query-friendly format based on a schema associated with structured data 12S and / or an extraction schema 202 associated with semi-structured data 12U. Here, ingestion refers to obtaining and / or importing user data 12 into storage system 140 (e.g., into data warehouse 142) to allow the system to use the ingested user data 12 (e.g., through query system 150). Generally, data can be ingested in real time, where manager 200 imports data as it is emitted from a source (e.g., user 10 or user device 110 of user 10) or in batches of discrete data blocks imported by manager 200 at periodic time intervals.

[0032] In order to extract patterns from the semi-structured data 12U, the manager 200 receives user data 12, which includes the semi-structured data 12U, from the user 10. For example, Figure 1 The description describes a user 10 transmitting a number N files to a manager 200 for storage in a query-efficient format in a storage system 140. Here, a first file A includes structured data 12, 12S, while a second file B includes semi-structured data 12U (e.g., JSON data). When user data 12 includes unstructured data 12U, user 10 typically provides the manager 200 with some indication 14 indicating that user data 12 includes semi-structured data 12U. This indication 14 indicates to the manager 200 that the semi-structured data 12U fails to include a fixed pattern (e.g., a pattern similar to structured data 12S). In some implementations, when user data 12 is a mixture of both structured data 12S and semi-structured data 12U, the indication 14 may additionally identify the start and / or end points of the semi-structured data 12U. User 10 can use a user interface to coordinate communication between user data 12 and the manager 200 and storage system 140. Here, the user interface may include fields or graphical components that, when specified (or selected), indicate that user data 12 includes semi-structured data 12U. In some examples, manager 200 is able to detect that certain portions of the user data 12 it receives are semi-structured data 12U. That is, manager 200 can determine that the format of at least some of the user data 12 corresponds to semi-structured data 12U. For example, manager 200 recognizes that user data 12 includes JSON data and knows that JSON data is a common unstructured message / file format. Based on indication 14 that user data 12 includes semi-structured data 12U, manager 200 generates extraction pattern 202 for semi-structured data 12U during user data ingestion.

[0033] refer to Figure 2A-2H The manager 200 is configured to generate an extraction mode 202 for semi-structured data 12U, and facilitates the storage of the semi-structured data 12U according to the extraction mode 202. When the manager 200 receives an indication 14 that the semi-structured data 12U fails to include a fixed mode, the manager 200 resolves the semi-structured data 12U into multiple data paths 210, 210a-n. (See reference) Figure 2A Semi-structured data 12U corresponds to a JSON payload with a single transaction record. Although semi-structured data 12U (such as JSON data) can include several records from a single message or file, for ease of interpretation... Figure 2A Describe a single record.

[0034] Based on the text of the JSON payload, manager 200 identifies five data paths 210, 210a-e of the semi-structured data 12U. In some examples, when the semi-structured data 12U is JSON data, each data path 210 corresponds to a representation of a key-value or key-attribute structure within the semi-structured data 12U. For example, in the first data path 210a, the key is the text "component" and the value 212 is the text "sproket". Similarly, the second data path 210b has a key corresponding to the text "cost" and a value 212 corresponding to the number "23". Figure 2A It is also shown that when parsing semi-structured data 12U such as JSON data, the manager 200 may encounter keys with nested values ​​or attributes. Here, the manager 200 can identify each nested key-value pair as a separate data path 210. For example, based on the text of the JSON payload, “manufacturer” includes three attributes: “name”, “location”, and “company_id”. The first attribute, “manufacturer.name”, causes the manager 200 to generate a third path 210c. The second attribute, “manufacturer.location”, causes the manager 200 to generate a fourth data path 210d, and the third attribute, “manufacturer.company_id”, causes the manager 200 to generate a fifth data path 210e. In this example, each of these paths 210 is associated with a specific value 212, as shown by the value 212 connected to each specific data path line.

[0035] refer to Figure 2B-2GIn some implementations, the manager 200 uses each of the data paths 210 to generate an extracted pattern 202. For example, each key of a corresponding data path 210 becomes a separate column pattern 202, where the column stores the value 212 corresponding to that key. That is, once the manager 200 specifies a separate column pattern 202, the manager 200 performs the ingestion of the semi-structured data 12U by storing the value 212 from the record corresponding to the column pattern 202 in the semi-structured data 12U. Thus, the column value associated with a row entry (e.g., record 1) corresponds to the value 212 of the data path 210 that has been extracted to form the column pattern 202. Here, storing the value 212 of the data path 210 in columnar storage in this way can bring storage efficiency, especially when combined with storage compression techniques (e.g., run-length encoding) or other compatible data storage techniques on similar types of data.

[0036] In some examples, the manager 200 generates the extracted pattern 202 by additionally considering the data type 220 of the parsed text. Here, the manager 200 considers the data type 220 because the data type 220 of the semi-structured data 12U can dynamically change between records. To account for these changes when constructing the pattern 202, the manager 200 can identify the data types included in the semi-structured data 12U. For example, the manager 200 determines the data type 220 associated with a particular data path 210. The data type 220 can also indicate when a data path 210 differs from or has become different from another data path 210. That is, for example, the data type 220 of a first value 212 associated with a first data path 210 can change when compared to the data type 220 of a second value 212 associated with the first data path 210 (e.g., between the first and second records). Due to this change in data type 220, the manager 200 can store the first value 212 in a different column of the pattern 202 with a column that is different from the second value 212. By adopting this method, the manager 200 is able to ensure that data type changes occurring from record to record do not impair the efficient storage of queries on the semi-structured data 12U. For illustration, if the first record includes a manufacturing name as a string, and the second record includes a manufacturing name as an integer, then a query 160 requesting the value 212 corresponding to the manufacturing name will result in a response 162 that may include both a string and an integer. Here, the user 10 may receive a query response 162 containing both a string and an integer, and not understand how the integer is a valid result of the query 160 for the manufacturing name. Since this situation is feasible due to the nature of the semi-structured data 12U, the manager 200 can extract the data type 220 to facilitate additional constraints on the extracted pattern 202. For example, additional constraints on the extracted pattern 202 could be minimum / maximum statistics specific to a particular data type 220. In other words, strings can have minimum / maximum statistics different from those of integers (i.e., numbers). Therefore, these additional constraints or statistics can be used to trim portions of the data during read operations (e.g., at the file granularity). In some examples, the manager 200 allows the user 10 to have any type of messy data in the data path 210, such that the column or data path 210 can be a mixture of data types 220 (e.g., strings and integers), so that the user 10 can read the data he or she ingests as is, regardless of the data type 220 corresponding to the value 212.

[0037] Figure 2B-2G This is an example depicting how the manager 200 slices or columnarizes the values ​​212 from the semi-structured data 12U into individual columns constrained by the extracted pattern 202. Here, as... Figure 2C-2GAs shown in more detail, the extracted pattern 202 also constrains which values ​​212 can be stored in which columns based on data type 220. Figure 2C In this context, the first data path 210a corresponding to "component" becomes a first column pattern 202, 202a with a first data type 220, 220a as a string. Therefore, the manager 200 stores the first value 212a from the first record "sprocket" as a string and the second value 212b from the second record "wheel" as a string as entries in a column with the first column pattern 202a. Figure 2D In this context, the second data path 210b corresponding to "cost" becomes a second column pattern 202, 202b with second data types 220, 220b, where the second data type is an integer and a floating-point number. Here, the manager 200 stores the third value 212c "23" and the fourth value 212d "700.23" as entries in a column with the second column pattern 202b, where both the third and fourth values ​​are integers (e.g., "23") or floating-point numbers (e.g., "700.23"). Figure 2E In this example, the third data path 210c corresponding to "manufacturer.name" becomes a third column pattern 202, 202c with third data types 220, 220c as strings. In this example, the manager 200 stores the fifth value 212e "Sprocket Labs" and the sixth value 212f "Roundabout," both of which are strings, as entries in a column with the third column pattern 202c. Figure 2F In this context, the fourth data path 210d corresponding to "manufacturer.location.address" becomes a fourth column pattern 202, 202d with a fourth data type 220, 220d as an array. Here, the manager 200 stores the values ​​222h-m corresponding to the array from each record, which identifies the address of the manufacturer specified by the fourth column pattern 202d. The column includes the array length, which indicates how many times the value 212 is repeated in its parent length. For example, len = {3, 3} specifies that the first "address" has 3 values, and the second "address" also has 3 values. Figure 2GIn this example, the fifth data path 210e corresponding to "manufacturer.company_id" becomes a fifth column pattern 202, 202e with a fifth data type 220e as an integer. In this example, the manager 200 stores the fourteenth value 212n "91724" and the fifteenth value 212o "83724", both of which are integers, as entries in a column with the fifth column pattern 202e.

[0038] In some examples, manager 200 can be configured to serialize value 212. Serialization generally refers to the process of converting value 212 into a given representation (e.g., a string representation) or from a given representation to value 212 (e.g., sometimes called deserialization). Therefore, when serialized data (e.g., the resulting bit sequence) is reread in the context of a serialization format, the serialized data can be reconstructed into its original pre-serialized form. Here, manager 200 can be configured to be compatible with a wide variety of serialization formats (e.g., including formats specific to JSON data). As an example, in the case where data path 210 has an integer data type 220 in the first row (e.g., the first record) and a string data type 220 in another row (e.g., the second record), manager 200 can serialize one or both of these values ​​220. By serializing value 212, manager 200 can facilitate consistent query processing of the stored value 212.

[0039] Figure 2F The diagram also demonstrates that the manager 200 can retain potentially important information from the semi-structured data 12U when storing it in the storage system 140. For example, the manager 200 includes row entries where the value 212 is explicitly null (e.g., value 212g). For example, explicit null (null) refers to a JSON null value. That is, the manager 200 has specific placeholders for null values ​​and even explicitly declares that the entry value 212 is null. In contrast, some storage technologies may not store explicit null values ​​or may ignore them for storage purposes. However, ignoring or removing explicit null values ​​can lead to downstream problems where keys with null values ​​are meaningful information. In some applications, it may be meaningful to know that a value does not exist (i.e., is null) rather than that the value is missing (i.e., may exist, but is unknown). For example, when using JSON data, the existence of JSON keys within a JSON object is meaningful, and therefore, preserving null values ​​is important. Similar to the explicit null value 212, the manager 200 can also store semi-structured data 12U such that the corresponding column value 212 of the column includes an empty array (i.e., an array indicating the format used for the value, but the array currently has no value).

[0040] Figure 2HTable 204, which describes storage system 140, can be a single large table 204, where each column corresponds to the extracted pattern 202, and each row is a record from one or more files / messages from user data 12. In other words, Figures 2A to 2G Each of Table 204 can be integrated together to form Figure 2H Table 204 is shown. Or, as shown in Table 204. Figure 2H The larger table 204 shown can be partitioned into smaller tables 204 (e.g., columnar slices), so that query processing does not have to evaluate the large table, but can evaluate a smaller table that more efficiently determines the answer to query 160 than the large table.

[0041] Using the extracted pattern approach, manager 200 can shard the semi-structured data 12U during ingestion in a manner similar to that of structured data 12S, making user data 12 ready for query processing during query runtime. By sharding and storing the semi-structured data 12U and structured data 12S in a similar manner, user data 12 can be stored together (e.g., in the same table 204) and, in some cases, jointly fulfilled in query 160, regardless of its structure. Furthermore, by columnarizing user data 12, manager 200 ensures that semi-structured data 12U can have similar query and storage efficiency to structured data 12S. For example, the table storage scheme allows for columnar processing during query processing, where query planning 152 does not need to read more data than is necessary to fulfill query 160. That is, query processing can use the extracted pattern 202 to filter or compute query response 162 at a data level more closely aligned with the query itself. For example, query system 150 uses the extracted pattern 202 to push down operations on user data 12 to reduce the amount of read and / or write operations that occur during query processing.

[0042] Figure 3This is a flowchart illustrating an example arrangement of operations for method 300 used to store semi-structured data 12U. At operation 302, method 300 receives user data 12 from user 10 of query system 150, wherein user data 12 includes semi-structured user data 12U. At operation 304, method 300 also receives an indication 14 that the semi-structured user data 12U fails to include a fixed pattern. In response to indication 14 that the semi-structured user data 12U fails to include a fixed pattern, method 300 performs operation 306, with two sub-operations 306, 306a-b. At sub-operation 306a, method 300 resolves the semi-structured user data 12U into multiple data paths 210. At sub-operation 306b, method 300 extracts the data type 220 associated with each corresponding data path 210 in the multiple data paths 210. At operation 308, method 300 stores semi-structured user data 12U as row entries in table 204 of a database communicating with query system 150, wherein each column value 212 associated with a row entry corresponds to a corresponding data path 210 among a plurality of data paths 210 and a data type 220 associated with the corresponding data path 210.

[0043] Figure 4 This is a schematic diagram of an example computing device 400 that can be used to implement the systems (e.g., remote system 130, storage system 140, query system 150, and / or manager 200) and methods (e.g., method 300) described in this document. The computing device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0044] Computing device 400 includes a processor 410 (e.g., data processing hardware), a memory 420 (e.g., memory hardware), a storage device 430, a high-speed interface / controller 440 connected to the memory 420 and a high-speed expansion port 450, and a low-speed interface / controller 460 connected to a low-speed bus 470 and the storage device 430. Each of components 410, 420, 430, 440, 450, and 460 is interconnected using various buses and may be suitably mounted on a common motherboard or otherwise mounted. Processor 410 can process instructions for execution within computing device 400, including instructions stored in memory 420 or on storage device 430, to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 480 coupled to high-speed interface 440. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be suitably used. In addition, multiple computing devices 400 can be connected, each of which provides a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0045] Memory 420 non-transitory stores information within computing device 400. Memory 420 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 420 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0046] Storage device 430 provides mass storage for computing device 400. In some embodiments, storage device 430 is a computer-readable medium. In various embodiments, storage device 430 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configuration. In other embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0047] High-speed controller 440 manages bandwidth-intensive operations for computing device 400, while low-speed controller 460 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 440 is coupled to memory 420, display 480 (e.g., via a graphics processor or accelerator), and high-speed expansion port 450, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 460 is coupled to storage device 430 and low-speed expansion port 470. Low-speed expansion port 470, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or network devices such as switches or routers, for example, via a network adapter.

[0048] The computing device 400 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 400a or multiple times in a set of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0049] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0050] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0051] The processes and logical flows described in this specification can be executed by one or more programmable processors that perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuitry (e.g., FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit)). Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors and any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, or both, such as magnetic, magneto-optical, or optical discs. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by dedicated logic circuits or incorporated into dedicated logic circuits.

[0052] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0053] Various embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations including: User data is received from users of the query system, including semi-structured user data; Receiving the semi-structured user data does not include instructions for a fixed pattern; In response to the indication that the semi-structured user data does not include the fixed pattern: The semi-structured user data is parsed into multiple data paths; as well as Extract the data type associated with each of the plurality of data paths; as well as The semi-structured user data is stored as row entries in a table of a database that communicates with the query system. Each column value associated with the row entry corresponds to a specific data path among the plurality of data paths and the data type associated with that data path. The user data further includes structured user data with a corresponding fixed pattern, and The tables in the database include one or more row entries corresponding to the structured user data.

2. The method according to claim 1, wherein, The semi-structured user data includes JavaScript object representation JSON.

3. The method according to claim 2, wherein, The corresponding column associated with the row entry includes explicit null values.

4. The method according to claim 2, wherein, The corresponding column values ​​associated with the row entry include an empty array.

5. The method according to claim 1, wherein, The storage of the semi-structured user data as row entries in the table of the database further includes: Identify a first data type and a second data type associated with a first data path among the plurality of data paths parsed from the semi-structured user data; The first value of the semi-structured user data corresponding to the first data type of the first data path is stored in the first row entry of the first column of the table; and The second value of the semi-structured user data corresponding to the second data type of the first data path is stored in the second row entry of the first column of the table. The second data type is a different data type from the first data type.

6. The method according to claim 1, wherein, The operation further includes, during the query runtime: The user receives queries from the query system for data associated with the stored semi-structured user data; Determine the corresponding data path of the stored semi-structured user data in response to the query; as well as In response to the query, a query response is generated that includes the corresponding column values ​​of the row entries corresponding to the corresponding data path of the stored semi-structured user data.

7. The method according to claim 1, wherein, The user data corresponds to the log data of the cloud computing resources associated with the user.

8. The method according to claim 1, wherein, Each of the plurality of data paths corresponds to a key in the key-value pair of the semi-structured user data.

9. The method according to any one of claims 1-8, wherein, The corresponding column values ​​for the row entries include nested arrays.

10. A system comprising: Data processing hardware; as well as Memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including the following: User data is received from users of the query system, including semi-structured user data; Receiving the semi-structured user data does not include instructions for a fixed pattern; In response to the indication that the semi-structured user data does not include the fixed pattern: The semi-structured user data is parsed into multiple data paths; as well as Extract the data type associated with each of the plurality of data paths; as well as The semi-structured user data is stored as row entries in a table of a database that communicates with the query system. Each column value associated with the row entry corresponds to a specific data path among the plurality of data paths and the data type associated with that data path. The user data further includes structured user data with a corresponding fixed pattern, and The tables in the database include one or more row entries corresponding to the structured user data.

11. The system according to claim 10, wherein, The semi-structured user data includes JavaScript object representation JSON.

12. The system according to claim 11, wherein, The corresponding column associated with the row entry includes explicit null values.

13. The system according to claim 11, wherein, The corresponding column values ​​associated with the row entry include an empty array.

14. The system according to claim 10, wherein, The storage of the semi-structured user data as row entries in the table of the database further includes: Identify a first data type and a second data type associated with a first data path among the plurality of data paths parsed from the semi-structured user data; The first value of the semi-structured user data corresponding to the first data type of the first data path is stored in the first row entry of the first column of the table; and The second value of the semi-structured user data corresponding to the second data type of the first data path is stored in the second row entry of the first column of the table. The second data type is a different data type from the first data type.

15. The system according to claim 10, wherein, The operation further includes, during the query runtime: The user receives queries from the query system for data associated with the stored semi-structured user data; Determine the corresponding data path of the stored semi-structured user data in response to the query; as well as In response to the query, a query response is generated that includes the corresponding column values ​​of the row entries corresponding to the corresponding data path of the stored semi-structured user data.

16. The system according to claim 10, wherein, The user data corresponds to the log data of the cloud computing resources associated with the user.

17. The system according to claim 10, wherein, Each of the plurality of data paths corresponds to a key in the key-value pair of the semi-structured user data.

18. The system according to any one of claims 10-17, wherein, The corresponding column values ​​for the row entries include nested arrays.

Images