Efficient Storage and Query of Schema-less Data

Abstract

A method (300) for storing semi-structured data (12U) includes receiving user data (12) including semi-structured user data from a user (10) of a query system (150). The method includes receiving an indication (14) that the semi-structured user data does not include a fixed schema. In response, the method further includes parsing the semi-structured user data into a plurality of data paths (210) and extracting a data type (220) associated with each respective data path of the plurality of data paths. The method further includes storing the semi-structured user data as row entries in a table (204) of a database in communication with the query system, each column value associated with the row entry corresponding to each of the plurality of data paths and the data type associated with each respective data path.

Description

Technical Field

[0001] This disclosure relates to efficient storage and query of schema - less data.

Background Art

[0002] Background Currently, as applications generate a significant amount of data, query systems and other analysis tools are continuously evolving to assist in data analysis. Further, if users cannot analyze their data in an efficient and / or cost - effective manner, the value of generating a vast amount of user data may be significantly reduced. To ensure that users can analyze their data, query - processing systems have begun to operate in conjunction with data - storage systems. User data can be stored in a storage structure that, by collaborating, facilitates analysis operations such as queries or other analyses. Unfortunately, the data - storage structures that facilitate those operations are often limited in their ability to support semi - structured or schema - less data.

Summary of the Invention

Means for Solving the Problems

[0003] Summary One aspect of the present disclosure provides a computer-executable method for storing semi-structured data. When executed by data processing hardware, the method causes the data processing hardware to perform operations. The operations include receiving user data from a user of a query system, where the user data includes unstructured user data. The operations further include receiving an indication that the unstructured user data does not include a fixed schema. In response to the indication that the unstructured user data does not include a fixed schema, the operations further include parsing the unstructured user data into a plurality of data paths and extracting a data type associated with each respective data path of the plurality of data paths. The operations further include storing the unstructured user data as a row entry in a table of a database that communicates with the query system, where each column value associated with the row entry corresponds to each of the plurality of data paths and the data type associated with each respective data path.

[0004] Another aspect of the present disclosure provides a system capable of storing unstructured data. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving user data from a user of a query system, where the user data includes unstructured user data. The operations further include receiving an indication that the unstructured user data does not include a fixed schema. In response to the indication that the unstructured user data does not include a fixed schema, the operations further include parsing the unstructured user data into a plurality of data paths and extracting a data type associated with each respective data path of the plurality of data paths. The operations further include storing the unstructured user data as a row entry in a table of a database that communicates with the query system, where each column value associated with the row entry corresponds to each of the plurality of data paths and the data type associated with each respective data path.

[0005] Embodiments of the method or system of the present disclosure may include one or more of the following optional features. In some embodiments, the user data further includes structured user data having respective fixed schemas, and the database table includes one or more row entries corresponding to the structured user data. In some examples, the unstructured user data includes JavaScript (registered trademark) Object Notation (JSON). In these examples, each column associated with a row entry may include an explicit null value. In these examples also, each column value of each column associated with a row entry may include an empty array. In some configurations, storing unstructured user data as a row entry in a database table further includes identifying a first data type and a second data type associated with a first data path among a plurality of data paths parsed from the unstructured data. In these configurations, storing unstructured user data as a row entry in a database table further includes storing a first value of the unstructured user data corresponding to the first data type of the first data path in a first row entry of a first column of the table, and storing a second value of the unstructured user data corresponding to the second data type of the first data path in a second row entry of the first column of the table. Here, the second data type is a data type different from the first data type.

[0006] In some examples, the operations for a method or system also include, during query execution time, receiving from a user of the query system a query on data associated with stored unstructured data, and in response to the query, determining each data path of the stored unstructured data, and in response to the query, generating a query response that includes each column value of a row entry corresponding to each data path of the stored unstructured data. The user data may correspond to log data for cloud computing resources associated with the user. Each data path of the plurality of data paths may correspond to a key for a key-value pair of unstructured user data. Each column value of the row entry may include a nested array.

[0007] Details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the specification and drawings, as well as from the claims.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 2E

Figure 2F

Figure 2G

Figure 2H

Figure 3

Figure 4

DETAILED DESCRIPTION OF THE INVENTION

[0009] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION A data storage system may store user or client data in one or more large queryable tables. The overall structure of a table includes data in some form of individual records organized into rows called row entries. The length of a row of data can vary based on the table schema and / or the number of columns or fields associated with a particular record (i.e., row). A table schema refers to a specified format of a table that can define column names (e.g., fields), the data type of a particular column, and / or other information. That is, the schema of a table is a completeness constraint imposed on the table that defines how the table is organized. For example, a table schema constrains the type or form of data in a particular column for one or more row entries. As an example, the schema for a particular column of a table is a name field that is a string, and the value of the row entry for that column is a name having a string data type.

[0010] In some embodiments, the storage system is configured to generate a table schema based on the attributes of the user data received by the storage system. For example, the storage system receives user data in a row-oriented format according to a specific schema before ingestion. In other embodiments, the user or client operates in cooperation with the storage system to define the schema of the user data prior to any transfer of the user data to the storage system. In either of these cases, the ability to query the stored user data depends on how the user data is stored within the data storage system. Thus, when the data of a record is stored as a row entry in a table based on the table schema, the table schema affects query processing. For this reason, structured data that refers to data including a specified schema can facilitate ingestion and enable efficient read and / or write operations on the data ingested during query processing. For example, each row of a table corresponds to a single order form from a parts supplier, while the columns of the table include schemas such as the name of the parts supplier (i.e., the manufacturer name), the purchased parts, the part cost, the address of the supplier (i.e., the manufacturer), and the supplier identifier (i.e., the company identifier). When the storage system receives a new order form record, the storage system arranges the data within that record into the appropriate columns of the table. By this type of arrangement, the storage system can respond to a query from the query system by planning one or more actions (e.g., a read operation or a write operation) using the table schema. By way of illustration, a customer may use a query system that functions in cooperation with the storage system to inquire about which order forms have occurred within a specific data range. In response to this query, the query processing generates a query plan that sorts the order forms by date and then filters out all dates that do not correspond to the specific data range specified by the query using a filter. The query processing then returns the results of the sorting and filtering as a response to the query.In other words, here, the query plan performed by query processing enhances the schema of the table with the order forms in order to efficiently (e.g., with minimal latency) satisfy the query.

[0011] Unfortunately, the reality is that user data can take any form. What this means is that while structured user data with a specified schema is preferred to facilitate ingestion and / or query processing, user data can also be unstructured or semi-structured data. Here, unstructured or semi-structured data refers to data that does not have a fixed schema. For example, unstructured data can refer to data that has no schema at all or is completely without a schema, while semi-structured data can refer to data that has a partial lack of schema. Thus, the term unstructured or semi-structured data refers to data where some part of the data does not have a fixed schema. This unstructured or semi-structured data can also be referred to as "schema-less" data because the structure of the data can vary between entries (i.e., records). Thus, from a storage perspective, it is not always easily apparent how to organize a table of multiple records resulting from schema-less data. Due at least in part to the dynamic and non-fixed nature of the data that is at least partially unstructured, this unstructured data has become a cause of problems for query systems and underlying storage systems. For example, in the absence of a fixed schema, the ingestion process can be difficult to columnarize the at least partially unstructured data into a table in an efficient manner for query processing.

[0012] What complicates this problem is that the client or user may not want to specify a particular schema for the unstructured data portion. For example, the client may not have confidence in its own ability to generate a meaningful schema from the perspective of query processing and / or data storage. Additionally, or alternatively, the user data may correspond to data automatically generated for the user (e.g., machine-generated data). As an example, the user may use a cloud computing environment, and the cloud computing environment may report the state of the user's data (or resources) by generating log files. In this situation, asking the user to generate a schema for this form of user data (e.g., user data log files) may be a burden for the user or may exceed the user's capabilities. In other words, the user may not know which relevant schema should be specified for those types of files. Therefore, in these scenarios, the query system and / or storage system requires a user-independent approach.

[0013] Furthermore, unstructured or semi-structured data is increasing in popularity because specific unstructured data formats are easy to create. This is particularly true in machine-generated contexts when the data is easily automatically generated in some part of its unstructured form (e.g., automatically generated log files). One such popular data format is JavaScript Object Notation (JSON). JSON data often functions as a data interchange format and is an increasingly common schema-less file format. JSON is partially language-independent, which means it is popular because it is compatible with all programming languages. Additionally, JSON is a text-based and lightweight format, making it easily generatable whether generated by a user or a machine. JSON data has some form of text structure and key-value or attribute-value notation, but the structure is not fixed between records. Since JSON data lacks a consistent structure, efficiently querying JSON data is somewhat difficult because query processing cannot use a formal schema in query plan creation or query validation.

[0014] In particular, when using JSON, there are several approaches to query processing for JSON data. In one approach, some systems interpret the entire JSON file as a string. This can store the data of the JSON file, but presents problems for the query system because much of the efficiency of the query system (e.g., query speed) relates to a fixed structure or schema. In other words, a fixed schema associated with the data entries enables the query system to create an efficient query plan. Part of the efficiency for query plan creation and query execution is that the entire data entry does not need to be read during query processing. However, when the storage system stores the JSON file as a string, queries about specific entries or records in the JSON file generally force the query processing to handle not just a portion of the entry but the entire string. That is, query processing generally needs to read the entire string to identify the relevant portion for the response to the query. In this sense, the string-based storage approach for JSON data has been considered less than ideal for the query system.

[0015] To address some of the above problems caused by data when at least some of the parts are unstructured (e.g., unstructured or semi-structured data), the techniques described herein promote efficient storage and / or retrieval of such data (e.g., unstructured or semi-structured data) by extracting a schema from the data. Although this technique is generally referred to below with respect to semi-structured data, the techniques described in detail can be used for any data that is wholly (e.g., unstructured data) or partially schema-less. By extracting a schema from semi-structured data, this technique also avoids relying on a user during schema generation for semi-structured data. Here, for storage and query purposes, the operation during ingestion processes semi-structured data, partitions it into a columnar format, and extracts a schema from the semi-structured data. Based on the extracted schema, when the semi-structured data is stored in a storage system, if a query requests only a portion of the semi-structured data, the extracted schema causes query processing to occur only at the portion requested by the query, generating a query result (also called a query response). For example, if a query requests a portion of semi-structured data in a particular column of a row entry, the query processing uses the extracted schema to read only the values in that particular column rather than reading the entire semi-structured data (or a larger portion thereof).

[0016] This extraction method can also provide the further advantage that both semi-structured and structured data can be captured by the storage system during the capture process. In other words, during capture, the storage system captures a single blob of user data per row, including both structured and semi-structured data. The ability to process both forms of data can be beneficial to the user, as it eliminates the need to initiate two different capture processes (i.e., one for structured data and one for semi-structured data). For example, a record with structured data can have fields that are semi-structured so that the storage system can store the unstructured part of the data in the same table as the structured data. In this regard, query results can return responses that include data from both structured and semi-structured data, thereby making the query process more robust across different data structures.

[0017] FIG. 1 is a diagram illustrating an example of a data management environment 100. A user device 110 associated with a user 10 generates user data 12 during the execution of its computing resources 112 (e.g., data processing hardware 114 and / or memory hardware 116). For example, user 10 generates user data 12 using an application running on data processing hardware 114 of user device 110. Since various applications have the ability to generate user data 12, user 10 often utilizes other systems (e.g., remote system 130, storage system 140, or query system 150) for the storage and / or management of user data.

[0018] In some examples, the user device 110 is a local device (e.g., associated with the location of user 10) that uses its own computing resources 112 to communicate (e.g., via network 120) with one or more remote systems 130. Additionally or alternatively, the user device 110 utilizes its access to remote resources (e.g., remote computing resources 132) to run applications for user 10. User data 12 generated through the use of user device 110 is first stored locally (e.g., in data storage 118 of memory hardware 116) and then transmitted to remote system 130 at the time of creation or sent to remote system 130 through network 120. For example, user device 110 uses remote system 130 to transmit user data 12 to storage system 140.

[0019] In some examples, user 10 utilizes computing resources 132 of remote system 130 (e.g., a cloud computing environment) for storage of user data 12. In these examples, remote system 130 may receive user data 12 when it is generated by various user applications (e.g., streaming data). Here, a data stream (e.g., of user data 12) refers to a continuous or generally continuous feed of data arriving at remote system 130 for storage and / or further processing. In some configurations, instead of streaming user data 12 continuously to remote system 130, user 10 and / or remote system 130 are configured to send user data 12 in batches (e.g., at intermittent frequencies) for processing. Remote system 130 is very similar to user device 110 but includes computing resources 132 such as remote data processing hardware 134 (e.g., servers and / or CPUs) and memory hardware 136 (e.g., disks, databases, or other forms of data storage).

[0020] In some configurations, the remote computing resources 132 are resources utilized by various systems associated with and / or communicating with the remote system 130. As shown in FIG. 1, these systems can include a storage system 140 and / or a query system 150. In some examples, the functionality of these systems 140, 150 can be integrated with each other in different permutations (e.g., can be built on top of each other) or can be separate systems having the ability to communicate with each other. For example, the storage system 140 and the query system 150 can be combined (e.g., as shown by the dashed line surrounding the system in FIG. 1) into a single system. The remote system 130 with the computing resources 132 can be configured to host one or more functions of those systems 140, 150. In some embodiments, the remote system 130 is a distributed system in which the computing resources 132 are distributed across one or more locations accessible via the network 120.

[0021] In some examples, the storage system 140 is configured to operate a data warehouse 142 (e.g., a data store and / or one or more databases) as data storage means for the user 10 (or users). Generally speaking, the data warehouse 142 can be designed to store data from one or more sources and to analyze, report, and / or integrate the data from those sources. With the data warehouse 142, a user (e.g., an organizational user) can have a central storage repository and a storage data access point. By including the user data 12 in a central repository such as the data warehouse 142, the data warehouse 142 can simplify data acquisition for functions such as data analysis and / or data reporting (e.g., by the query system 150 and / or the analysis system). Further, the data warehouse 142 can be configured to store a significant amount of data so that the user 10 (e.g., an organizational user) can store a large amount of historical data to understand data trends. If the data warehouse 142 can be the primary or sole data storage repository for the user's data 12, the storage system 140 may often receive large amounts of data (e.g., gigabytes / second, terabytes / second, or more) from the user device 110 associated with the user 10. Additionally or alternatively, as the storage system 140, the storage system 140 and / or the storage warehouse 142 can be configured for multiple users (e.g., multiple employees of an organization) from a single data source and / or for data security (e.g., data redundancy) for concurrent multi-user access. In some configurations, the data warehouse 142 is persistent and / or non-volatile so that data is not overwritten or erased by new incoming data by default.

[0022] The query system 150 is configured to request information or data from the storage system 140 in the form of a query 160. In some examples, the query 160 is initiated by the user 10 as a request for user data 12 within the storage system 140 (e.g., a data export request). For example, the user 10 interacts with the query system 150 (e.g., an interface associated with the query system 150) to retrieve user data 12 stored in the data warehouse 142 of the storage system 140. Here, the query 160 may be user-initiated (i.e., directly requested by the user 10) or system-initiated (i.e., configured by the query system 150 itself). In some examples, the query system 150 configures a routine or repeating query 160 (e.g., at some specified frequency) that enables the user 10 to perform analysis or monitoring of the user data 12 stored in the storage system 140.

[0023] The format of query 160 can be various, but generally includes a reference to specific user data 12 stored in storage system 140. Here, query system 150 operates in conjunction with storage system 140 such that storage system 140 stores user data 12 according to a specific schema, and query system 150 can generate query 160 based on information regarding the specific schema. For example, data storage system 140 stores user data 12 in a table format where user data 12 is input into rows and columns of a table. In the case of the table format, the user data 12 within the table has rows and columns corresponding to the schema associated with user data 12. For example, user data 12 may refer to an order form created by user 10. In this example, one or more tables storing user data 12 may include columns for purchased parts, costs of the parts, manufacturers of the purchased parts, and details regarding the manufacturers (e.g., manufacturer address and / or company identifier of the manufacturer). Here, each row corresponds to a record or entry (e.g., order form), and each column value associated with the row entry in the columns of the table corresponds to a specific part of the schema. Since storage system 140 can receive user data 12 according to a specific schema (e.g., the schema indicated by schema manager 200), storage system 140 is configured to store user data 12 such that query system 150 can access elements of that format (e.g., relationships, headings, or other schema) associated with user data 12 (e.g., providing further context or definition for user data 12).

[0024] In response to query 160, query system 150 generates a query response 162 that satisfies, or attempts to satisfy, the request of query 160 (e.g., a request for specific user data 12). Generally speaking, query response 162 includes the user data 12 requested by query system 150 in query 160. Query system 150 may return this query response 162 to the entity that issued query 160 (e.g., user 10) or to another entity or system that communicates with query system 150. For example, query 160 itself or query system 150 may specify that query system 150 transmit one or more query responses 162 to a system associated with user 10, such as an analysis system. For example, user 10 uses an analysis system to perform analysis on user data 12. In many cases, query system 150 is set up to generate routine queries 160 regarding user data 12 in storage system 140 so that the analysis system can perform its analysis (e.g., at a specific frequency). For example, query system 150 executes a daily query 160 that retrieves the user log data for the past seven days that the analysis system analyzes and / or represents.

[0025] In some embodiments, when query system 150 receives an input for query 160, query system 150 is configured to determine a query plan 152 to execute query 160. In other words, query 160 often refers to generally basic-level tables without specific reference to the actual structure of the tables in storage system 140. For example, to simply state, query 160 queries the table of user data 12 in storage system 140 to export purchase data for manufacturer Sprocket Labs over the past six months. The input format of query 160 is simplified for ease of use as a user interface for extracting from more complex tables and / or storage structures of user data 12 in storage system 140. Thus, user 10 who executes or writes query 160 need not know the actual storage structure. Rather, to generate query 160, it is only necessary to know the schema or fields of the table structure at a higher level. Query system 150, in combination with storage system 140, decomposes query 160 from user 10 and rewrites query 160 into a form that identifies potential operators (e.g., read / write operations) regarding user data 12 to perform query 160 regarding the underlying structure of user data 12. That is, when query system 150 receives query 160, it takes in query 160 and plans how to execute that query 160 against the actual structure of storage system 140. This plan creation may require identification of a subset of the table (e.g., a partition) and / or files regarding the table corresponding to query 160.

[0026] Referring to FIGS. 1 and 2A-2H, data management environment 100 also includes a schema manager 200 (also referred to as manager 200). Manager 200 is configured to manage schema extraction during user data ingestion. Here, schema extraction refers to the process of generating a schema for semi-structured user data 12, 12U during ingestion and storage (e.g., by storage system 140). Schema extraction generally occurs during ingestion, but can occur prior to ingestion as a preprocessing step depending on the configurational requirements of the larger system. Manager 200 performs schema extraction because, in contrast to structured user data 12, 12S, semi-structured user data 12U does not include a fixed schema. In the absence of a fixed schema, manager 200 instead extracts a schema for semi-structured user data 12U, whereby user data 12 (e.g., both structured data 12S and semi-structured data 12U) is segmented and made storable in storage system 140. Schema extraction by manager 200 occurs when user data 12 is actually loaded into storage system 140 (e.g., during ingestion), whereby manager 200 specifies how user data 12 is to be stored in one or more tables 204 of storage system 140. Manager 200 can extract a schema for semi-structured user data 12U by performing and / or coordinating operations (e.g., storage operations and / or query operations) related to systems 140, 150 for user 10. For example, manager 200 analyzes “in-progress” semi-structured user data 12U because it is being processed and encoded during ingestion (e.g., to become a lower-level storage component). In this regard, characteristics of the semi-structured data (e.g., data path and / or data type) are considered by manager 200 when determining how and where to store / encode the semi-structured data.That is, the manager 200 does not necessarily establish a schema for the semi-structured data 12U as a preprocessing step before ingestion and apply that schema. Instead, it interprets the schema for the semi-structured data 12U when ingestion occurs. For example, this means that there may not always be a clear schema.

[0027] The functionality of the manager 200 may be centralized (e.g., it may be present in one of the systems 140, 150) or distributed across the systems 140, 150 according to its design. In some examples such as FIG. 1, the manager 200 is configured to receive user data 12 from the user 10 and facilitate storage operations in the storage system 140. For example, the manager 200 facilitates data load requests by the user 10. In response to a load request by the user 10, the manager 200 can ingest the user data 12 and further convert the user data 12 into a query-suitable format based on the schema associated with the structured data 12S and / or the extracted schema 202 associated with the semi-structured data 12U. Here, ingestion refers to obtaining the user data 12 and / or importing it into the storage system 140 (e.g., the data warehouse 142), thereby making the ingested user data 12 available for use by (e.g., the query system 150). Generally speaking, data can be ingested in real time when the manager 200 imports the data when it is transmitted from a source (e.g., the user 10 or the user device 110 of the user 10), or in batches where the manager 200 imports discrete data chunks at periodic time intervals.

[0028] To extract a schema from semi-structured data 12U, the manager 200 receives user data 12 including the semi-structured data 12U from the user 10. For example, FIG. 1 illustrates that the user 10 transmits N files, which are some numbers, to the manager 200, and those files are stored in the storage system 140 in a query-efficient format. Here, the first file A includes structured data 12, 12S, while the second file B includes semi-structured data 12U (e.g., JSON data). When the user data 12 includes unstructured data 12U, the user 10 generally provides the manager 200 with some indications 14 that the user data 12 includes semi-structured data 12U. This indication 14 indicates to the manager 200 that the semi-structured data 12U does not include a fixed schema (e.g., the schema of the structured data 12S, etc.). In some embodiments, when the user data 12 is mixed data including both structured data 12S and semi-structured data 12U, the indication 14 may additionally identify where the semi-structured data 12U starts and / or ends. The user 10 operates in cooperation with the transmission of the user data 12 to the manager 200 and the storage system 140 using a user interface. Here, the user interface may include a field or some graphic component that indicates that the user data 12 includes semi-structured data 12U when specified (or selected). In some examples, the manager 200 can detect that some part of the received user data 12 is semi-structured data 12U. That is, the manager 200 may determine that the format of at least a part of the user data 12 corresponds to the semi-structured data 12U. For example, the manager 200 recognizes that the user data 12 includes JSON data and knows that JSON data is a common unstructured message / file format. Based on the indication 14 that the user data 12 includes semi-structured data 12U, the manager 200 generates an extracted schema 202 for the semi-structured data 12U during the ingestion of the user data 12.

[0029] Referring to FIGS. 2A-2H, the manager 200 is configured to generate an extracted schema 202 for the semi-structured data 12U and to facilitate storage of the semi-structured data 12U according to the extracted schema 202. When the manager 200 receives an indication 14 that the semi-structured data 12U does not include a fixed schema, the manager 200 parses the semi-structured data 12U into a plurality of data paths 210, 210a-210n. Referring to FIG. 2A, the semi-structured data 12U corresponds to a JSON payload having a single transaction record. Although semi-structured data 12U, such as JSON data, can include several records in a single message or file, FIG. 2A illustrates a single record for ease of explanation.

[0030] Based on the text of the JSON payload, the manager 200 identifies five data paths 210, 210a - 210e of the semi - structured data 12U. In some examples, when the semi - structured data 12U is JSON data, each data path 210 corresponds to the representation of a key - value or key - attribute structure within the semi - structured data 12U. For example, in the first data path 210a, the key is the text "component" and the value 212 is the text "sprocket". Similarly, the second data path 210b has a key corresponding to the text "cost" and a value 212 corresponding to the number "23". FIG. 2A also shows that while parsing semi - structured data 12U such as JSON data, the manager 200 may encounter keys with nested values or attributes. Here, the manager 200 can identify each nested key - value pair as a separate data path 210. For example, according to the text of the JSON payload, "manufacturer" includes three attributes: "name", "location", and "company_id". The first attribute, "manufacturer.name", causes the manager 200 to generate a third data path 210c. The second attribute, "manufacturer.location", causes the manager 200 to generate a fourth data path 210d, and the third attribute, "manufacturer.company_id", causes the manager 200 to generate a fifth data path 210e. In this example, each of these paths 210 is associated with a specific value 212 as indicated by the value 212 connected to each specific data path line.

[0031] Referring to FIGS. 2B - 2G, in some embodiments, the manager 200 uses each of the data paths 210 to generate the extracted schema 202. For example, each key of each data path 210 becomes an individual column schema 202, and the column stores the value 212 corresponding to that key. That is, when the manager 200 designates an individual column schema 202, the manager 200 imports the semi - structured data 12U by storing the value 212 from the record within the semi - structured data 12U corresponding to that column schema 202. Thus, the column values associated with a row entry (e.g., Record 1) correspond to the values 212 of the data path 210 extracted to form the column schema 202. Here, storing the values 212 of the data path 210 in column - oriented storage by this approach results in storage efficiency, particularly when combined with storage compression techniques (e.g., run - length encoding) or other affinity - related data storage techniques for similar data types.

[0032] In some examples, the manager 200 generates the extracted schema 202 by further considering the data type 220 of the parsed text. Here, since the semi-structured data 12U may dynamically change the data type 220 between records, the manager 200 considers the data type 220. To account for the above changes when constructing the schema 202, the manager 200 may identify the data types included in the semi-structured data 12U. For example, the manager 200 determines the data type 220 associated with a particular data path 210. The data type 220 may also indicate when one data path 210 is different from or has become different from another data path 210. That is, for example, the data type 220 of the first value 212 associated with the first data path 210 may change when compared to the data type 220 of the second value 212 associated with the first data path 210 (e.g., between the first record and the second record). Due to this change in the data type 220, the manager 200 may store the first value 212 in a different column with a column schema 202 different from that of the second value 212. By using this approach, the manager 200 may be able to ensure that data type changes occurring between records do not compromise query-efficient storage for the semi-structured data 12U. For example, if the first record contains a manufacturing name that is a string and the second record contains a manufacturing name that is an integer, a query 160 requesting the value 212 corresponding to the manufacturing name may result in a response 162 that includes both a string and an integer. Here, the user 10 may receive a query response 162 having both a string and an integer and may not understand how the integer is a valid result of the query 160 for the manufacturing name. Since this situation is possible due to the nature of the semi-structured data 12U, the manager 200 may extract the data type 220 that contributes to additional constraints on the extracted schema 202. For example, the additional constraints on the extracted schema 202 may be minimum / maximum statistics specialized for a particular data type 220. In other words, a string may have different minimum / maximum statistics from an integer (i.e., a number).Accordingly, these additional constraints or statistics can be used to discard a portion of the data during the read operation (e.g., at the file granularity). In some examples, this functionality of the manager 200 enables the user 10 to read their imported data as is, regardless of the corresponding data type 220, such that the column or data path 210 can be a mix of data types 220 (e.g., strings and integers), enabling the user 10 to have all kinds of messy data in the data path 210.

[0033] Figures 2B - 2G are examples illustrating how manager 200 subdivides or column - orientates the values 212 from semi - structured data 12U into individual columns constrained by the extracted schema 202. Here, as shown in more detail by FIGS. 2C - 2G, the extracted schema 202 also constrains, based on the data type 220, which values 212 can be stored in which columns. In FIG. 2C, the first data path 210a corresponding to "component" becomes the first column schemas 202, 202a having the first data types 220, 220a which are strings. Accordingly, manager 200 stores the first value 212a "sprocket" from the first record which is a string and the second value 212b "wheel" from the second record which is a string as entries in the column having the first column schema 202a. In FIG. 2D, the second data path 210b corresponding to "cost" becomes the second column schemas 202, 202b having the second data types 220, 220b which are integer and float. Here, manager 200 stores the third value 212c "23" and the fourth value 212d "700.23", both of which are either integer (e.g., "23") or float (e.g., "700.23"), as entries in the column having the second column schema 202b. In FIG. 2E, the third data path 210c corresponding to "manufacturer.name" becomes the third column schemas 202, 202c having the third data types 220, 220c which are strings. In this example, manager 200 stores the fifth value 212e "Sprocket Labs" and the sixth value 212f "Roundabout", both of which are strings, as entries in the column having the third column schema 202c. In FIG. 2F, the fourth data path 210d corresponding to "manufacturer.location.address" becomes the fourth column schemas 202, 202d having the fourth data types 220, 220d which are arrays. Here, manager 200 stores the values 222h - 222m corresponding to the arrays from each record identifying the manufacturer's address as specified by the fourth column schema 202d. This column includes the array length which indicates how many times the value 212 is repeated in the length of its parent.For example, len = {3,3} specifies that the first "address" has three values and the second "address" also has three values. In FIG. 2G, the fifth data path 210e corresponding to "manufacturer.company_id" becomes the fifth column schema 202, 202e having the fifth data type 220e which is an integer. In this example, the manager 200 stores as entries in the column having the fifth column schema 202e the fourteenth value 212n "91724" and the fifteenth value 212o "83724", both of which are integers.

[0034] In some examples, the manager 200 may be configured to serialize the value 212. Serialization generally refers to the process of converting the value 212 to a given representation (e.g., a textual representation) or from a given representation (which may also be called deserialization in some cases). Thus, when the serialized data (e.g., the resulting bit sequence) is read again in the context of the serialization format, the serialized data can be reconstructed into its original pre-serialized form. Here, the manager 200 may be configured to be compatible with a wide range of serialization formats (including formats specialized for JSON data, for example). As an example, if the data path 210 has an integer data type 220 for the first row (e.g., the first record) and a string data type 220 for another row (e.g., the second record), the manager 200 may serialize one or both of those values 220. By serializing the value 212, the manager 200 can facilitate consistent query processing of the stored value 212.

[0035] FIG. 2F also shows that the manager 200 can store potentially important information from the semi-structured data 12U when storing the semi-structured data 12U in the storage system 140. For example, the manager 200 includes row entries where the value 212 is an explicit null (e.g., value 212g). For example, an explicit null refers to JSON null. That is, the manager 200 has a specific placeholder for null and further states that the entry value 212 is null. On the other hand, some storage technologies may not store explicit nulls or may ignore explicit nulls for storage purposes. Further, ignoring or removing explicit nulls can cause downstream problems when keys with null values are meaningful information. In some applications, it may be meaningful to know that a value does not exist (i.e., is null) rather than that a value is missing (i.e., may exist but is unknown). For example, in the case of JSON data, the presence of a JSON key within a JSON object has significance, and thus it is important to store its null value. Similar to the explicit null value 212, the manager 200 can also store the semi-structured data 12U such that each column value 212 of the column includes an empty array (i.e., the array is in the form for its value but indicates that there is no currently presented value for that array).

[0036] FIG. 2H illustrates that the table 204 of the storage system 140 can be a single large table 204, each column corresponding to an extracted schema 202, and each row being a record from one or more files / messages of the user data 12. In other words, each table 204 of FIGS. 2A - 2G can be integrated with each other to form the table 204 shown in FIG. 2H. Alternatively, a large table 204 as shown in FIG. 2H can be split into small tables 204 (e.g., column-oriented slices) such that the answer to the query 160 can be verified more efficiently by evaluating small tables rather than having to evaluate the large table.

[0037] In the case of the extracted schema approach, so that the user data 12 can be made query - process - executable during query execution time, the manager 200 can, during ingestion, similarly subdivide the semi - structured data 12U into structured data 12S. By similarly subdividing and storing the semi - structured data 12U and the structured data 12S, the user data 12 can be stored together regardless of structure (e.g., can be stored in the same table 204), and, depending on the situation, can jointly satisfy the query 160. Further, by the manager 200 column - orienting the user data 12, the manager 200 ensures that the semi - structured data 12U can have similar query and storage efficiency with respect to the structured data 12S. For example, a table storage schema enables column - oriented processing during query processing, and the query plan 152 does not need to read more data than the amount necessary to satisfy the query 160. That is, the query processing can filter or calculate the query response 162 at the data level that aligns closer to the query itself using the extracted schema 202. For example, the query system 150 uses the extracted schema 202 to push down operations on the user data 12 to reduce the amount of read and / or write operations that occur during query processing.

[0038] Figure 3 is a flowchart of an exemplary configuration of operations for a method 300 of storing semi-structured data 12U. In operation 302, method 300 receives user data 12 from a user 10 of query system 150, where user data 12 includes semi-structured user data 12U. In operation 304, method 300 also receives an indication 14 that the semi-structured user data 12U does not include a fixed schema. In response to the indication 14 that the semi-structured user data 12U does not include a fixed schema, method 300 performs an operation 306 with two sub-operations 306a-306b. In sub-operation 306a, method 300 parses the semi-structured user data 12U into a plurality of data paths 210. In sub-operation 306b, method 300 extracts a data type 220 associated with each respective data path 210 of the plurality of data paths 210. In operation 308, method 300 stores the semi-structured user data 12U as a row entry in a table 204 of a database that communicates with query system 150, and each column value 212 associated with that row entry corresponds to each of the plurality of data paths 210 and the respective data type 220 associated with each data path 210.

[0039] Figure 4 is a schematic diagram of an exemplary computing device 400 that can be used to implement the systems (e.g., remote system 130, storage system 140, query system 150, and / or manager 200) and methods (e.g., method 300) described in this document. Computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0040] The computing device 400 includes a processor 410 (e.g., data processing hardware), a memory 420 (e.g., memory hardware), a storage device 430, a high-speed interface / controller 440 connected to the memory 420 and the high-speed expansion port 450, and a low-speed interface / controller 460 connected to the low-speed bus 470 and the storage device 430. The components 410, 420, 430, 440, 450, and 460 are interconnected using various buses and may be mounted on a common motherboard or in other suitable manners as appropriate. The processor 410 can process instructions for execution within the computing device 400, including instructions stored in the memory 420 or on the storage device 430 for displaying graphical information for a graphical user interface (GUI) on an external input / output device such as a display 480 coupled to the high-speed interface 440. In other embodiments, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 400 may be connected, and each device may provide a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0041] Memory 420 stores information non - transiently within computing device 400. Memory 420 may be a computer - readable medium, a volatile memory unit, or a non - volatile memory unit. The non - transient memory 420 is a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 400. Examples of non - volatile memory include, but are not limited to, flash memory and read - only memory (ROM) / programmable read - only memory (PROM) / erasable programmable read - only memory (EPROM) / electrically erasable programmable read - only memory (EEPROM) (e.g., typically used for firmware such as a boot program). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase - change memory (PCM), and disks or tapes.

[0042] Storage device 430 can provide a mass storage device for computing device 400. In some embodiments, storage device 430 is a computer - readable medium. In various different embodiments, storage device 430 may be a number of devices including, but not limited to, a floppy (registered trademark) disk device, a hard disk device, an optical disk device, or a tape device, flash memory or other similar solid - state memory devices, or a device or other configuration within a storage area network. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine - readable medium such as memory 420, storage device 430, or memory - on - processor 410.

[0043] While the high-speed controller 440 manages bandwidth-intensive operations for the computing device 400, the low-speed controller 460 manages lower-bandwidth-intensive operations. Such task distribution is merely exemplary. In some embodiments, the high-speed controller 440 is coupled to a high-speed expansion port 450 that can receive a memory 420, a display 480 (e.g., through a graphics processor or accelerator), and various expansion cards (not shown). In some embodiments, the low-speed controller 460 is coupled to a storage device 430 and a low-speed expansion port 470. The low-speed expansion port 470, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, wireless Ethernet), may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0044] As shown in the figure, the computing device 400 may be implemented in several different forms. For example, this may be implemented as a standard server 400a, or multiple times as a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0045] The various embodiments of the systems and techniques described herein can be implemented in digital electronics and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device, which can be of specific or general purpose.

[0046] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0047] The processes and logical flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be performed by special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit). A processor suitable for the execution of a computer program includes, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read only memory or a random access memory or both. Indispensable elements of a computer are a processor for performing instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also be operatively coupled to one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, to receive data from, or transfer data to, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0048] To provide interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, and optionally, a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input received from the user can be in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending a document to and receiving a document from a device used by the user, such as by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0049] Some embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are also within the scope of the following claims.

Claims

1. A method executed by a computer, which, when executed by data processing hardware, receives user data including semi-structured user data from a user of a query system; receives an indication that the semi-structured user data does not include a fixed schema; in response to the indication that the semi-structured user data does not include the fixed schema, parses the semi-structured user data into a plurality of data paths; extracts a data type associated with each of the plurality of data paths; causes the data processing hardware to store the semi-structured user data as a row entry in a table of a database communicating with the query system, each column value associated with the row entry corresponds to each of the plurality of data paths and the data type associated with each of the data paths; the operations are during query execution time receiving, from the user of the query system, a query for data associated with the stored semi-structured user data; determining, in response to the query, each data path of the stored semi-structured user data; further including generating, in response to the query, a query response including each column value of the row entry corresponding to each of the data paths of the stored semi-structured user data.

2. The method according to claim 1, wherein the semi-structured user data includes JavaScript Object Notation (JSON).

3. The method according to claim 2, wherein each column associated with the row entry includes an explicit null value.

4. The method according to claim 2, wherein each column value of each column associated with the row entry includes an empty array.

5. Storing the semi-structured user data as the row entry in the table of the database includes: identifying a first data type and a second data type associated with a first data path among the plurality of data paths parsed from the semi-structured user data; storing a first value of the semi-structured user data corresponding to the first data type of the first data path in a first row entry of a first column of the table; and further including storing a second value of the semi-structured user data corresponding to the second data type of the first data path in a second row entry of the first column of the table, wherein the second data type is a data type different from the first data type, the method according to claim 1.

6. The method according to claim 1, wherein the user data corresponds to log data about cloud computing resources associated with the user.

7. The method according to claim 1, wherein each data path of the plurality of data paths corresponds to a key for a key-value pair of the semi-structured user data.

8. The method according to claim 1, wherein each column value of the row entry includes a nested array.

9. Data processing hardware, and Memory hardware communicating with the data processing hardware, wherein when executed on the data processing hardware, receiving user data including semi-structured user data from a user of a query system; receiving an indication that the semi-structured user data does not include a fixed schema; In response to the indication that the semi-structured user data does not include the fixed schema, parsing the semi-structured user data into a plurality of data paths, extracting a data type associated with each of the plurality of data paths, storing instructions in the data processing hardware to cause the data processing hardware to perform operations including storing the semi-structured user data as a row entry in a table of a database that communicates with the query system, each column value associated with the row entry corresponds to each of the plurality of data paths and each of the data types associated with each of the data paths, the operations include, during query execution time, receiving, from the user of the query system, a query for data associated with the stored semi-structured user data, determining, in response to the query, each data path of the stored semi-structured user data, generating, in response to the query, a query response including each column value of the row entry corresponding to each of the data paths of the stored semi-structured user data, a system.

10. The system according to claim 9, wherein the semi-structured user data includes JavaScript Object Notation (JSON).

11. The system according to claim 10, wherein each column associated with the row entry includes an explicit null value.

12. The system according to claim 10, wherein each column value of each column associated with the row entry includes an empty array.

13. storing the semi-structured user data as the row entry in the table of the database includes Identifying a first data type and a second data type associated with a first data path among the plurality of data paths parsed from the semi-structured user data; Storing a first value of the semi-structured user data corresponding to the first data type of the first data path in a first row entry of a first column of the table; Further comprising storing a second value of the semi-structured user data corresponding to the second data type of the first data path in a second row entry of the first column of the table, The system according to claim 9, wherein the second data type is a data type different from the first data type.

14. The system according to claim 9, wherein the user data corresponds to log data about cloud computing resources associated with the user.

15. The system according to claim 9, wherein each data path of the plurality of data paths corresponds to a key for a key-value pair of the semi-structured user data.

16. The system according to claim 9, wherein each column value of the row entry includes a nested array.

Images