Dynamic performance tuning based on implied data characteristics

Abstract

The present disclosure relates to dynamic performance tuning based on implied data characteristics. Techniques for improving system performance based on data characteristics are disclosed. A system can receive updates to a first data set at a first frequency. The system selects a first storage configuration for storing the first data set from a plurality of storage configurations based on the first frequency, and stores the first data set according to the first storage configuration. The system can also receive updates to a second data set at a second frequency. The system selects a second storage configuration for storing the second data set from the plurality of storage configurations based on the second frequency, and stores the second data set according to the second storage configuration. The second storage configuration is different from the first storage configuration.

Description

[0001] This application is a divisional application of the PCT application filed on September 4, 2020, with national application number 202080105124.X and invention title "Dynamic Performance Tuning Based on Implicit Data Characteristics," which has entered the Chinese national phase. Technical Field

[0002] This disclosure relates to data analysis. More specifically, this disclosure relates to techniques for improving system performance based on data characteristics. Background Technology

[0003] Internet of Things (IoT) devices generate datasets that include three different categories of data: identifier data, use case data, and runtime data.

[0004] Identifier data can include human-readable and / or human-understandable characteristics of the device that generates the data. Specifically, identifier data can include aspects such as device name, serial number, purchase information, and similar characteristics. Identifier data can include many different identifiers for the device providing the data. Typically, identifier data includes data that remains constant or changes very little. Identifier data will have the lowest cardinality in the system (e.g., the fewest different fields or combinations of fields). Use case data typically refers to data on the various ways in which the device providing the data is used. For example, for a thermostat, use case data could include "heating" or "cooling". Similar to identifier data, use case data is typically represented as one or more strings. Runtime data can include information measured when the device is used in the field. As an example, for a thermostat, runtime data could include temperature measurements and timestamps indicating the time the temperature was recorded. Runtime data is typically the largest data category, often exceeding the data volume by several orders of magnitude.

[0005] The methods described in this section are possible methods, but not necessarily methods that have been previously conceived or adopted. Therefore, unless otherwise stated, no method described in this section should be assumed to be prior art simply because it is included in this section. Attached Figure Description

[0006] The embodiments are illustrated in the accompanying drawings by way of example rather than limitation. It should be noted that references to "a" or "an" embodiment in this disclosure do not necessarily refer to the same embodiment, and they mean at least one. In the accompanying drawings: Figure 1 The illustration shows a system according to one or more embodiments; Figure 2 The illustration shows an example set of operations for improving system performance based on data characteristics, according to one or more embodiments; Figure 3A and Figure 3B The illustration shows an example system for improving system performance based on data characteristics; and Figure 4 A block diagram illustrating a computer system according to one or more embodiments is shown. Detailed Implementation

[0007] In the following description, numerous specific details are set forth for purposes of explanation, in order to provide a comprehensive understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in different embodiments. In some examples, well-known structures and devices are described with reference to block diagrams to avoid unnecessarily obscuring the invention.

[0008] 1. General Overview

[0009] 2. Architecture Overview

[0010] 3. Improve system performance based on data characteristics

[0011] 4. Example Implementation

[0012] 5. Other; Extension

[0013] 6. Hardware Overview

[0014] 1. General Overview

[0015] One or more embodiments include systems for performance tuning based on data characteristics. Users of big data systems may use heterogeneous storage mechanisms that do not support traditional cost-based query tuning because they are supported by storage systems that do not provide commonly used features such as indexes, row counts, data cardinality statistics, and similar tools, or because the heterogeneous nature of the storage systems hinders cross-storage optimization. However, for time-series data, inherent aspects of the data, storage patterns, and implicit aspects of the data based on the query itself can be used to infer assumed patterns and help optimize query execution. User-defined metadata about how data sources are associated provides insight into what kinds of data are stored in each system, and metadata can be used to infer missing statistics about the stored data. Similarly, once some information about a particular data source is known, characteristics of other data sources can be assumed.

[0016] In one embodiment, the system can determine how to store and access data by grouping it. For example, when processing data from one or more Internet of Things (IoT) devices, the system can group the data into one of three data families that it considers to have the characteristics of IoT data. The system can use information about which data has already been grouped into each family to presume which data should be found in the other families and adjust the storage and retrieval processes accordingly.

[0017] For example, three data families for IoT data could be identifier data, use case data, and runtime data. Systems can use these data families because they allow for efficient, definable classification, and because these classifications, in turn, allow for the estimation of use cases.

[0018] Data fields within a dataset can be grouped based on the characteristics of the data. In particular, data fields can be categorized based on one or more characteristics of the data stored in that field, such as cardinality (e.g., the number of entries in the data field) and / or update frequency (e.g., how often the data in the field is updated)).

[0019] One or more embodiments described in this specification and / or stated in the claims may not be included in this general overview section.

[0020] 2. Architecture Overview

[0021] Figure 1 A system 100 according to one or more embodiments is illustrated. Figure 1 As shown, system 100 includes a performance tuning engine 102, a user interface 114, one or more external data sources 118, a data repository 120, and various components thereof. In one or more embodiments, system 100 may include a performance tuning engine 102, a user interface 114, one or more external data sources 118, a data repository 120, and various components thereof. Figure 1 The components shown may have more or fewer components. Figure 1 The components shown can be local to each other or geographically distant. They can be implemented using software and / or hardware. Figure 1 The components are shown in the diagram. Each component can be distributed across multiple applications and / or machines. Multiple components can be combined into one application and / or machine. An operation described for one component can be performed alternatively by another component.

[0022] In one or more embodiments, user interface 114 refers to hardware and / or software configured to facilitate communication between a user and interface creation engine 102. User interface 114 can be used by a user accessing an interface (e.g., a dashboard interface) for work and / or personal activities. User interface 114 can be associated with one or more devices (such as display 116, including monitors, televisions, projectors, and / or similar devices) for presenting visual media. User interface 114 presents user interface elements and receives input via these elements. Examples of interfaces include graphical user interfaces (GUIs), command-line interfaces (CLIs), haptic interfaces, and voice command interfaces. Examples of user interface elements include checkboxes, radio buttons, drop-down lists, list boxes, buttons, toggles, text fields, date and time pickers, command lines, sliders, pages, and tables.

[0023] In this embodiment, different components of the user interface 114 are specified using different languages. The behavior of user interface elements is specified using a dynamic programming language (such as JavaScript). The content of user interface elements is specified using a markup language (such as Hypertext Markup Language (HTML) or XML User Interface Language (XUL)). The layout of user interface elements is specified using a stylesheet language (such as Cascading Style Sheets (CSS)). Alternatively, the user interface 114 is specified using one or more other languages ​​(such as Java, C, or C++).

[0024] In one or more embodiments, performance tuning engine 102 refers to hardware and / or software configured to perform the operations described herein for dynamic performance tuning based on implicit data characteristics. Reference hereinafter refers to... Figure 2 An example of operations for dynamic performance tuning based on implicit data characteristics is described.

[0025] In one embodiment, the performance tuning engine 102 includes a data classification component 104. The data classification component 104 may refer to hardware and / or software configured to perform the operations described herein for identifying input from a user, including operations that can be incorporated by reference.

[0026] In one embodiment, the performance tuning engine 102 includes a category analysis component 106. The category analysis component 106 may refer to hardware and / or software configured to perform query suggestion generation and / or identification.

[0027] In one embodiment, the performance tuning engine 102 includes a storage configuration manager 108. The storage configuration manager 108 may refer to selected hardware and / or software configured to execute a selected query.

[0028] In this embodiment, one or more components of the performance tuning engine 102 utilize the machine learning engine 110. Machine learning encompasses various techniques in the field of artificial intelligence, which deal with computer-implemented, user-independent processes for solving problems with variable inputs.

[0029] In some embodiments, machine learning engine 110 trains machine learning model 112 to perform one or more operations. Specifically, machine learning engine 110 may train machine learning model 112 to receive one or more records of a dataset as input and partition the one or more records into one or more data fields. One or more machine learning engines 110 may enable machine learning model 112 to determine one or more characteristics associated with each data field and select a storage configuration for the data field based on the one or more characteristics.

[0030] Training machine learning model 112 uses training data to generate a function that computes a corresponding output given one or more inputs to machine learning model 112. In at least some embodiments, machine learning model 112 may include a two-step process, such that data is input into an algorithm of the output model, and data is fed into the model to produce an output. This output may correspond to a prediction based on previous machine learning. In embodiments, the output includes labels, classifications, and / or categorizations assigned to the provided inputs. Machine learning model 112 corresponds to a learned model for performing a desired operation, such as labeling, classifying, and / or categorizing the inputs. Performance tuning engine 102 may use multiple machine learning engines 110 and / or multiple machine learning models 112 for different purposes.

[0031] In embodiments, machine learning engine 110 may use supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and / or another training method or a combination thereof. In supervised learning, the labeled training data includes input / output pairs, where each input is labeled with a desired output (also referred to as a supervision signal) (e.g., label, classification, and / or categorization). In semi-supervised learning, some inputs are associated with a supervision signal, while others are not. In unsupervised learning, the training data does not include a supervision signal. Reinforcement learning uses a feedback system, where machine learning engine 110 receives positive and / or negative reinforcement as it attempts to solve a specific problem (e.g., optimize performance in a specific scenario according to one or more predetermined performance criteria). In embodiments, machine learning engine 110 initially uses supervised learning to train machine learning model 112, and then uses unsupervised learning to continuously update machine learning model 112.

[0032] In embodiments, the machine learning engine 110 may use a variety of different techniques to label, classify, and / or categorize the input. The machine learning engine 110 may transform the input into a feature vector describing one or more properties (“features”) of the input. The machine learning engine 110 may label, classify, and / or categorize the input based on the feature vector. Alternatively or additionally, the machine learning engine 110 may use clustering (also known as cluster analysis) to identify commonalities in the input. The machine learning engine 110 may group the input (i.e., cluster) based on these commonalities. The machine learning engine 110 may use hierarchical clustering, k-means clustering, and / or another clustering method or a combination thereof. In embodiments, the machine learning engine 110 includes an artificial neural network. The artificial neural network includes multiple nodes (also known as artificial neurons) and edges between the nodes. Edges may be associated with corresponding weights that represent the strength of the connection between nodes, which the machine learning engine 110 adjusts as machine learning progresses. Alternatively or additionally, the machine learning engine 110 may include a support vector machine. A support vector machine represents the input as a vector. Machine learning engine 110 can label, classify, and / or categorize the input based on the vector. Alternatively or additionally, machine learning engine 110 can use a Naive Bayes classifier to label, classify, and / or categorize the input. Alternatively or additionally, given a specific input, the machine learning model can apply a decision tree to predict the output for that input. Alternatively or additionally, machine learning engine 110 can apply fuzzy logic when labeling, classifying, and / or categorizing the input among a fixed set of mutually exclusive options is impossible or impractical. The foregoing machine learning model 112 and techniques are discussed for illustrative purposes only and should not be construed as limiting one or more embodiments.

[0033] In this embodiment, because the machine learning engine 110 applies different inputs to the machine learning model 112, the corresponding outputs are not always accurate. As an example, the machine learning engine 110 may use supervised learning to train the machine learning model 112. After training the machine learning model 112, if subsequent inputs are the same as those contained in the labeled training data, and the output is the same as the supervision signal in the training data, then the output is guaranteed to be accurate. If the inputs are different from those contained in the labeled training data, then the machine learning engine 110 may generate inaccurate or uncertainly accurate corresponding outputs. In addition to producing a specific output for a given input, the machine learning engine 110 may also be configured to generate indicators representing the confidence (or lack of confidence) in the accuracy of the output. Confidence indicators may include numerical ratings, Boolean values, and / or any other kind of indicator corresponding to the confidence (or lack of confidence) in the accuracy of the output.

[0034] In this embodiment, machine learning models can be used for anomaly detection. There are three main categories of anomaly detection techniques. Unsupervised anomaly detection techniques detect anomalies in an unlabeled test dataset by finding instances that seem to least conform to the rest of the dataset, under the assumption that most instances in the dataset are normal. Supervised anomaly detection techniques can provide a first dataset already labeled "normal" and a second dataset already labeled "abnormal" as training inputs, and use the first and second datasets to train a classifier to determine whether an unlabeled input should be labeled "normal" or "abnormal." Semi-supervised anomaly detection techniques build a model representing normal behavior from a given normal training dataset, and then test the likelihood of test instances generated by that model.

[0035] In this embodiment, the performance tuning engine 102 is configured to receive data from one or more external data sources 118. External data source 118 refers to hardware and / or software that operates independently of the performance tuning engine 102. For example, the hardware and / or software of external data source 118 may be under the control of a different entity (e.g., a different company or other type of organization) than the entity controlling the query suggestion engine. External data source 118 may supply data associated with one or more Internet of Things (IoT) devices.

[0036] In this embodiment, the performance tuning engine 102 is configured to retrieve data from the external data source 118 by "pulling" data via the application programming interface (API) of that external data source 118 using user credentials that the user has provided to that specific external data source 118. Alternatively or additionally, the external data source 118 may be configured to "push" data to the performance tuning engine 102 via the API using access keys, passwords, and / or other types of credentials that the user has provided to the external data source 118. The performance tuning engine 102 may be configured to receive data from the external data source 118 in many different ways.

[0037] In one or more embodiments, the data repository 120 is any type of storage unit and / or device (e.g., file system, database, collection of tables, or any other storage mechanism) used for storing data. Furthermore, the data repository 120 may include multiple different storage units and / or devices. These multiple different storage units and / or devices may or may not be of the same type or located in the same physical location. Additionally, the data repository 120 may be implemented or executed on the same computing system as the performance tuning engine 102 and / or the external data source 118. Alternatively or additionally, the data repository 120 may be implemented or executed on a computing system separate from the performance tuning engine 102 and / or the external data source 118. The data repository 120 may be coupled to the performance tuning engine 102 and / or the external data source 118 via a direct connection or via network communication. In some embodiments, the data repository 120 includes one or more data caches, allowing the system 100 to store data in a persistent but not permanent location. Alternatively or additionally, the system 100 may use at least a portion of the one or more data caches included in the data repository 120 to store summary statistics related to the dataset.

[0038] A data repository can serve as a store for many types of data. For example, data stored in a data repository can represent data received from one or more IoT devices. Data can be categorized based on various criteria. Categorization can be chosen to allow for efficient, definable data classification and to provide estimates of use cases. Without algorithmic classification, users with expertise in business space and computing technologies are required to manually declare and optimize the partitioning of data on disk, the creation of indexes for efficient retrieval, and caching strategies. As a specific example, data can be categorized as one of identifier data 122, use case data 124, or runtime data 126, such as... Figure 1 As shown in the diagram. Identifier data 122 may include data that identifies the specific device providing the data. Use case data 124 may include data associated with a specific function of the device providing the data. Runtime data 126 may include data associated with the activities performed by the device providing the data.

[0039] Identifier data 122 can be human-readable characteristics and / or human-understandable data characteristics. Identifier data 122 can include aspects such as device name, serial number, purchase data, and similar characteristics. Identifier data 122 can include many different identifiers of the device providing the data. Typically, identifier data 122 includes data that remains constant or changes very little. The system can assume that identifier data 122 will have the lowest cardinality in the system (e.g., the fewest distinct entries). The system can assume that identifier data 122 is typically root-level for grouping in many forms of reporting and predictive analytics. For example, a machine learning algorithm predicting the probability of car failure based on mileage might first group the data by car brand and model, and then look for trends within that brand and / or model. From a storage / retrieval perspective, the system can assume that most interactions will use operators such as "equals" rather than range operators. Because identifier data 122 is root-level data, it has fewer records than use case data 124 or runtime data 126. Partitioning the dataset by any value in identifier data 122 may not provide much benefit when reading, but will be highly valuable when analyzing the dataset. Caching identifier data 122 is generally very efficient because it is rarely edited. Identifier data 122 tends to include string values. The records are extensive, but the set is not large; therefore, memory space can often be saved by hashing the identifier data into a lookup table. When joining data across families, identifier data 122 is typically the right-hand side of the join.

[0040] Use case data 124 typically refers to data on the various ways in which the device providing the data is used. For example, for a thermostat, use case data 124 could include "heating" or "cooling". Similar to identifier data 122, use case data 124 is typically represented as one or more strings. However, use case data is often less easy to hash because each part of a use case record can be an important feature. Additionally, analysis of the dataset can focus on drilling down within use case data 124. Therefore, use case data 124 is often accessed in various ways and at various points in the analysis. Overall system performance can be improved by caching use case data 124, which is partitioned sequentially by each field. Although this caching process caches more data than a single partition, caching is likely to be valuable. Furthermore, because use case data 124 has a relatively low row count, the memory cost of caching use case data is not excessive. Since the system decides that use case data 124 should be cached, on-disk storage of use case data does not require particularly high read efficiency. Instead, storage can be space-efficient, aggressively compressing use case data 124.

[0041] Runtime data 126 may include information about a specific instance of identifier data when it is used in the field. For example, for a thermostat, runtime data may include a timestamp indicating the time the temperature was measured and recorded. The system may determine that runtime data 126 is the largest data set, typically orders of magnitude larger. The system may determine that at least one field in runtime data 126 is a timestamp and needs to declare a timestamp or look for one in the data. Regardless of the initial size of runtime data 126, the system may determine that runtime data 126 will eventually grow to the point that it is excessive for any form of in-memory data read. In heterogeneous data environments, such as those with both relational databases and large-capacity data stores, runtime data 126 may be data that the system determines should be stored in a large-capacity data storage device. Because runtime data 126 is typically predicted to be large and includes numerous reports from the same physical object (e.g., currently running equipment), the system can determine that runtime data 126 is typically accessed via a time-range scan (e.g., all records from last week, last month, or last year) or via a time-range scan combined with partition keys from use case data 124 and / or identifier data 122 (e.g., all records from last year grouped by thermostats in heating mode). The system can algorithmically optimize storage by determining, based on record timestamps, that older data can be moved to less efficient storage at a rate based on the frequency declared by the user-defined analytics operation (e.g., a user running an analytics analysis of data from the past week will not be as sensitive to the slowness of reading 5 years of data as a user whose analytics are defined by long-term trends).

[0042] In this embodiment, system 100 is implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device can refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablet computers, laptop computers, desktop computers, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, special-purpose hardware devices, hardware routers, hardware switches, hardware firewalls, hardware network address translation (NAT), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile phones, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management equipment, routers, switches, controllers, access points, and / or client devices.

[0043] 3. Improve system performance based on data characteristics

[0044] Figure 2The illustration depicts an example set of operations for improving system performance based on data characteristics, according to one or more embodiments. These operations can be modified, rearranged, or omitted. Figure 2 One or more operations are shown. Therefore, Figure 2 The specific sequence of operations shown should not be interpreted as limiting the scope of one or more embodiments.

[0045] In an embodiment, the system (e.g., one or more components of system 100) receives a dataset (operation 202). In some embodiments, the dataset may be received as a single unit. In other embodiments, the dataset may be received over time (e.g., intermittently or periodically). In some aspects, data may be received from one or more external devices. In particular, data may be received from connected devices, such as Internet of Things (IoT) devices. The Internet of Things (IoT) is a system of interconnected computing devices, machines, and digital machines equipped with unique identifiers and the ability to transmit data over a network without requiring human-to-human or human-to-computer interaction. For example, IoT devices may include building or home automation devices, medical and / or healthcare devices, transportation devices (e.g., vehicle-to-vehicle and / or in-vehicle communication, traffic control, logistics, vehicle safety and / or control), agricultural monitoring devices, and the like. Many types of devices can utilize the IoT architecture.

[0046] The received data may include one or more data fields (also referred to as a dataset). As an example, the received data may include timestamps, one or more identifiers identifying the device transmitting the data to the system, one or more data items indicating the device's usage patterns, and one or more measurements performed by the device. In some aspects, each data item received at the system may include entries from each data field. In other aspects, data items received at the system may include only those fields that have changed since the previous data item was received.

[0047] The system can determine a classification method for data fields in the dataset (operation 204). This classification method can be used for one or more data fields in the dataset. In some embodiments, the same classification method is used to classify each data field in the dataset. In other embodiments, different classification methods can be used to classify different data fields.

[0048] As an example, data fields can be assigned to categories based on the frequency with which data is received or updated in that field. For instance, data can be categorized based on the absolute frequency of receiving or updating data (e.g., the number of items corresponding to a specific data field received or updated per second, per minute, or per hour). As another example, data can be assigned to categories based on relative frequency (e.g., the number of items corresponding to a specific data field received or updated compared to other fields in the dataset).

[0049] As another example, data fields can be assigned to categories based on the number of records in the data within that field. For instance, data can be categorized based on the absolute number of records (e.g., the number of records corresponding to a specific data field that is less than 100, less than 1000, etc.). As yet another example, data can be assigned to categories based on the relative number of records (e.g., the number of records corresponding to a specific data field compared to other fields in the dataset).

[0050] Each field of the received data can be categorized based on a defined categorization method (operation 206). The categories to which data fields can be assigned may include identifier data, use case data, or runtime data. Identifier data may include data that identifies the specific device providing the data. Use case data may include data associated with a specific function or operating mode of the device providing the data. Runtime data may include data associated with activities and / or measurements performed by the device providing the data.

[0051] In some embodiments, classifying data may include selecting a first threshold number and a second threshold number. The second threshold number may be greater than the first threshold number. In some embodiments, the threshold number may be selected based on the data contained in the dataset. In other embodiments, the threshold number may be predetermined. In some embodiments, the second threshold is an order of magnitude larger than the first threshold. The first and second thresholds may be used to classify data fields.

[0052] As an example, data fields that do not exceed a first threshold can be classified as identifier data. Identifier data can be human-level data characteristics. Identifier data can include aspects such as device name, serial number, purchase information, and similar characteristics. Identifier data can include many different identifiers of the device providing the data. Typically, identifier data consists of data that remains unchanged or changes very little. Therefore, identifier data may have a relatively high cardinality in the system (e.g., a relatively large number of different entries) but may be updated with the lowest possible frequency.

[0053] As another example, data fields exceeding a first threshold but not exceeding a second threshold can be categorized as use case data. Use case data typically refers to data on the various ways in which the provided device is used. For example, for a thermostat, use case data could include "heating" or "cooling." Generally, use case data changes relatively infrequently, but more frequently than identifier data. Therefore, use case data has a relatively low cardinality. However, the cardinality of use case data is larger than that of identifier data. Similarly, use case data is updated at a relatively low frequency, but more frequently than identifier data.

[0054] As a third example, data fields exceeding both the first and second thresholds can be classified as runtime data. Runtime data can include information about a specific instance of the identifier data when it is used in the field. For example, for a thermostat, runtime data could include a timestamp indicating the time the temperature was measured and recorded. The system can determine that runtime data is the largest data set, typically several orders of magnitude larger. The system can determine that at least one field in the runtime data is a timestamp and needs to declare or search for a timestamp in the data. Regardless of the initial size of the runtime data, the system can determine that the runtime data will eventually grow to the point where it is excessive for data reads from any form of memory.

[0055] The system can determine the storage configuration for storing data fields based on the determined classification (operation 208). Configuring storage may include selecting the storage medium where the data fields will be stored. In an embodiment, the configuration data also includes specifying a specific layout of the data within the selected storage medium. Configuring storage may include selecting a specific data structure for storing data field entries. Additionally, the storage configuration may specify specific partitions of the database to improve database performance.

[0056] In this embodiment, the system may rely on machine learning algorithms to track user input regarding the analysis of the dataset. For example, the system may track data access operations initiated by the user (e.g., join operations, table scan operations, read operations, etc.). The system may rely on machine learning to determine patterns in these data access operations and select storage configurations in part based on these patterns. For example, when the vast majority of data access operations are for data received within the past week, the system may determine that data older than a week can be stored in a more space-efficient but less read-efficient manner.

[0057] For example, storage configuration can be selected in part based on the size of a data field. Storage configuration can also be selected based on how data fields are typically accessed. For example, the frequency of data access and the type of access performed can influence the selected configuration. The system can associate specific access types and frequencies with specific data categories, allowing the system to select a specific storage configuration for a data field based on its categorization. In some embodiments, configuring storage can include selecting a storage configuration from multiple storage configurations. Multiple storage configurations can be predetermined based on typical usage of one or more storage devices.

[0058] As a specific example, identifier data is often the root level for grouping in many forms of reporting and predictive analytics. From a storage / retrieval perspective, most interactions with identifier data can use operators such as "equals" rather than range operators. Identifier data has fewer records than use case data or runtime data. Therefore, partitioning the dataset by any value in the identifier data provides little benefit when reading, but will be highly valuable when analyzing the dataset. Caching identifier data is generally very efficient because it is rarely edited. Identifier data tends to include string values. Records are extensive, but the set is not large; therefore, memory space can often be saved by hashing the identifier data into a lookup table. When joining data across families, identifier data is often the right-hand side of the join.

[0059] As another example, use case data is typically represented as one or more strings. However, use case data is often less easy to hash because each part of a use case record can be an important feature. Additionally, analysis of the dataset can focus on drilling down within the use case data. Therefore, use case data is often accessed in various ways and at various points in the analysis. Overall system performance can be improved by caching the use case data, partitioned sequentially by each field. While this caching process caches more data than a single partition, the caching is likely to be valuable. Furthermore, because use case data has a relatively low row count, the memory cost of caching use case data is not excessive. Because the system decides that use case data should be cached, on-disk storage of use case data does not need to be particularly high read efficiency. Instead, storage can be space-efficient, aggressively compressing the use case data.

[0060] As another example, runtime data can be stored in large-capacity data storage devices. Because runtime data typically comprises large datasets containing different reports (often many reports) from the same physical object (e.g., currently running equipment), the system can determine whether to access runtime data typically through a time-range scan (e.g., all records from last week, last month, or last year) or through a time-range scan combined with partition keys from use case data or identifier data (e.g., all records from last year grouped by thermostats in heating mode, or all records from last year grouped by serial number). The system can algorithmically optimize storage by determining, based on record timestamps, that older data can be moved to less efficient storage devices at a rate of movement declared based on the frequency of user-defined analytics operations (e.g., a user running analytics on data from the past week will not be as sensitive to the slowness of reading 5 years of data as a user whose analytics are defined by long-term trends).

[0061] In some embodiments, after a storage configuration is selected, the system stores data (operation 210). Storing data may include storing data to the selected storage device. Storing data may include creating a data structure based on the selected storage configuration. In some embodiments, data storage may include selecting a new data configuration. For example, in response to changes in user access patterns, changes in data inflow patterns, or as knowledge about these patterns grows, selecting a new storage configuration (and storing data based on the newly selected storage configuration) is not only possible but also highly probable.

[0062] 4. Example Implementation

[0063] For clarity, detailed examples are described below. The components and / or operations described below should be understood as a particular example that may not be applicable to some embodiments. Therefore, the components and / or operations described below should not be construed as limiting the scope of any claims.

[0064] Figure 3A A connected thermostat device 302 is shown, which transmits data to system 304 for collection, as indicated by arrow 306. Specifically, device 302 may transmit identifier data 306a, use case data 306b, and runtime data 306c. Figure 3A As shown, the thickness of the arrows corresponds to the relative number of updates for each data category.

[0065] like Figure 3BAs shown, the dataset includes data fields "Device Identifier," "Device Mode," "Temperature," and a timestamp. As shown in the "Timestamp" and "Temperature" data fields, device 302 transmits a temperature measurement to system 304 once per second. For each measurement, the device also transmits an identifier (e.g., the "Device Identifier" data field). Additionally, the device transmits an indicator of its operating mode (e.g., the "Device Mode" data field). Figure 3B In the example shown, device "Device_01" switches from "cooling" mode to "heating" mode at time 12:00:00. A change in mode can indicate user interaction that changes the mode, a scheduled mode change, or other reasons for mode switching.

[0066] The system categorizes each data field as either identifier data, use case data, or runtime data. The system sets a first threshold to 1 and a second threshold to 5. The "Device Identifier" field has 1 update, which does not exceed either the first or second threshold. Therefore, the "Device Identifier" data field is categorized as identifier data. The "Device Mode" field has 2 updates, which exceeds the first threshold but not the second threshold. Therefore, the "Device Mode" data field is categorized as use case data. The "Temperature" and "Timestamp" fields each have 11 updates, which exceeds both the first and second thresholds. Therefore, the "Temperature" and "Timestamp" data fields are each categorized as runtime data.

[0067] After categorizing the data, the system can determine the storage configuration for each category. For identifier data (such as the "Device Identifier" data field), the system can select a space-efficient storage configuration, such as a heavily compressed one. The system can further recommend caching the identifier data for fast retrieval. For use case data (such as the "Device Mode" data field), the system can select a space-efficient storage configuration, such as a heavily compressed one. For runtime data (such as the "Temperature" and "Timestamp" data fields), the system can recognize that these data fields are likely to grow rapidly and determine a storage configuration that allows for efficient storage of large volumes of data. Furthermore, the system can structure the data storage in a way that improves efficiency when reading data. For example, data can be stored chronologically, as analysis of the data is likely to result in retrieval of data spanning a specific time range. Data from each field can be stored according to the selected storage configuration.

[0068] 5. Other; Extension

[0069] The embodiments are directed to a system having one or more devices, which include a hardware processor and are configured to perform any of the operations described herein and / or stated in any of the following claims.

[0070] In an embodiment, the non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause to perform any operation described herein and / or stated in any of the claims.

[0071] Any combination of the features and functions described herein may be used according to one or more embodiments. In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary depending on the implementation. Therefore, the specification and drawings should be viewed in an illustrative rather than restrictive sense. The sole and exclusive indication of the scope of the invention, and what the applicant intends to define as the scope of the invention, is the literal and equivalent scope of the claims generated in the specific form of the set of claims arising from this application, including any subsequent amendments.

[0072] 6. Hardware Overview

[0073] According to one embodiment, the technology described herein is implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute the technology, or may include digital electronic devices (such as one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or network processing units (NPUs) that are persistently programmed to execute the technology, or may include one or more general-purpose hardware processors programmed to execute the technology according to program instructions in firmware, memory, other storage devices, or combinations thereof. Such dedicated computing devices may also combine custom hardwired logic, ASICs, FPGAs, or NPUs with custom programming to implement the technology. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hardwired and / or program logic to implement the technology.

[0074] For example, Figure 4 This is a block diagram illustrating a computer system 400 on which embodiments of the present invention can be implemented. The computer system 400 includes a bus 402 or other communication mechanism for transmitting information, and a hardware processor 404 coupled to the bus 402 for processing information. For example, the hardware processor 404 may be a general-purpose microprocessor.

[0075] Computer system 400 also includes main memory 406, such as random access memory (RAM) or other dynamic storage devices, coupled to bus 402 for storing information and instructions to be executed by processor 404. Main memory 406 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 404. Such instructions, when stored in non-transitory storage media accessible to processor 404, make computer system 400 a dedicated machine customized to perform the operations specified in the instructions.

[0076] Computer system 400 also includes a read-only memory (ROM) 408 or other static storage device coupled to bus 402 for storing static information and instructions of processor 404. Storage device 410 (such as a disk or optical disk) is provided and coupled to bus 402 for storing information and instructions.

[0077] Computer system 400 can be coupled to display 412 (such as a cathode ray tube (CRT)) via bus 402 for displaying information to the computer user. Input device 414 (including alphanumeric and other keys) is coupled to bus 402 for transmitting information and command selections to processor 404. Another type of user input device is a cursor control 416, such as a mouse, trackball, or cursor arrow keys, for transmitting directional information and command selections to processor 404 and controlling cursor movement on display 412. Such input devices typically have two degrees of freedom on two axes (a first axis, such as the x-axis, and a second axis, such as the y-axis), which allows the device to specify a position in a plane.

[0078] Computer system 400 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic. This custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, combined with the computer system, enables computer system 400 to become a special-purpose machine, or to program computer system 400 as a special-purpose machine. According to one embodiment, computer system 400 performs the techniques herein in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406. Such instructions may be read into main memory 406 from another storage medium, such as storage device 410. Executing the sequence of instructions contained in main memory 406 causes processor 404 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used instead of or in combination with software instructions.

[0079] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile and / or volatile media. For example, non-volatile media include optical discs or magnetic disks, such as storage device 410. Volatile media include dynamic memory, such as main memory 406. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical medium with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cartridges, content-addressable memory (CAM), and tri-state content-addressable memory (TCAM).

[0080] Storage media differ from transmission media, but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including the conductors that form bus 402. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0081] Various forms of media can be used to carry one or more sequences of one or more instructions to processor 404 for execution. For example, instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 400 may receive data over the telephone line and use an infrared transmitter to convert the data into an infrared signal. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 402. Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. Instructions received by main memory 406 may optionally be stored on storage device 410 before or after execution by processor 404.

[0082] Computer system 400 also includes a communication interface 418 coupled to bus 402. Communication interface 418 provides bidirectional data communication coupled to network link 420 connected to local network 422. For example, communication interface 418 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing data communication connectivity to a corresponding type of telephone line. As another example, communication interface 418 may be a Local Area Network (LAN) card to provide data communication connectivity to a compatible LAN. A wireless link may also be implemented. In any such implementation, communication interface 418 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0083] Network link 420 typically provides data communication to other data devices via one or more networks. For example, network link 420 may provide a connection to host computer 424 or to data devices operated by Internet service provider (ISP) 426 via local network 422. ISP 426, in turn, provides data communication services via a worldwide packet data communication network now commonly referred to as the “Internet” 428. Both local network 422 and Internet 428 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals carrying digital data to and from computer system 400 via various networks, as well as signals on network link 420 and through communication interface 418, are example forms of transmission media.

[0084] Computer system 400 can send messages and receive data (including program code) via a network, network link 420, and communication interface 418. In the Internet example, server 430 can transmit application request code via the Internet 428, ISP 426, local network 422, and communication interface 418.

[0085] The received code may be executed by processor 404 when it is received, and / or stored in storage device 410 or other non-volatile storage device for later execution.

[0086] In the foregoing description, embodiments of the invention have been described with reference to numerous specific details that may vary depending on the implementation. Therefore, the description and drawings should be viewed in an illustrative rather than restrictive sense. The sole and exclusive indication of the scope of the invention, and what the applicant intends to define as the scope of the invention, is the literal and equivalent scope of the claims generated in the specific form of the set of claims arising from this application, including any subsequent amendments.

Claims

1. A method comprising: Receive a time series dataset for a first time interval, the time series dataset comprising multiple records associated with corresponding multiple consecutive second time intervals, each record comprising the same set of fields, the set of fields comprising a first plurality of instances of a first field and a second plurality of instances of a second field; Analyze a first plurality of instances of a first field across the plurality of records to determine a first number of values ​​stored in a first plurality of instances of a first field across the plurality of records; Based on a first quantity equal to or less than a first threshold: classify the first data stored in the first plurality of instances of the first field as corresponding to the first data type; Analyze a second plurality of instances of the second field across the plurality of records to determine a second number of values ​​stored in a second plurality of instances of the second field across the plurality of records; Based on the second quantity exceeding the first threshold: classify the second data stored in the second multiple instances of the second field as corresponding to the second data type; Based on classifying the first data into a first data type, the first data is stored according to the first storage configuration; as well as Based on classifying the second data into a second data type, the second data is stored according to a second storage configuration that is different from the first storage configuration.

2. The method according to claim 1, further comprising: Analyze a third plurality of instances of the third field across the plurality of records to determine a third number of distinct values ​​stored in a third plurality of instances of the third field across the plurality of records; as well as Based on a second threshold where the third quantity exceeds a first threshold: the third data stored in the third multiple instances of the third field is classified as corresponding to the third data type.

3. The method of claim 2, wherein the first data type is name data associated with the device name. The second data type is use case data that describes the operating status of the device. The third data type includes runtime data of measurements generated by the device.

4. The method of claim 1, wherein the first threshold is 1, such that determining that the first quantity is equal to or less than the first threshold includes determining that multiple values ​​stored in a first plurality of instances of the plurality of records in the first field remain unchanged.

5. The method of claim 1, wherein analyzing a first plurality of instances of the first field across the plurality of records to determine a first number of values ​​stored in the first plurality of instances of the first field across the plurality of records comprises: Determine the number of times the value stored in the first plurality of instances of the first field changes between one of the plurality of records and a record that is temporally adjacent. Analyzing a second plurality of instances of the second field across the plurality of records to determine a second number of values ​​stored in the second plurality of instances of the second field across the plurality of records includes determining the number of times the values ​​stored in the second plurality of instances of the second field change between one of the plurality of records and time-adjacent records.

6. The method of claim 1, wherein the first storage configuration and the second storage configuration comprise at least one of the following: Hash the values ​​used for the corresponding data fields into a lookup table; The values ​​for the first and second fields are stored in separate storage devices corresponding to different data retrieval speeds; A first set of values ​​for a first plurality of instances of a first field and a second set of values ​​for a second plurality of instances of a second field are stored in separate storage devices with different data storage capacities. Specific data partitioning scheme; as well as The compression ratio used to compress data stored in the first and second fields.

7. The method of claim 1, wherein storing second data for a specific record according to a second storage configuration includes storing the second data together with the timestamp of the specific record from the time series dataset. The first data for storing the specific record according to the first storage configuration includes storing the first data without storing the timestamp of the corresponding record from the time series dataset.

8. A method comprising: Training machine learning The model classifies values ​​from time-series data, and the training includes: Obtain the training dataset; each training dataset contains historical data and includes: A time-series dataset corresponding to a specific time interval, the time-series dataset comprising: a first plurality of instances of a first field of a first plurality of records corresponding to the corresponding first plurality of time intervals; and For each record, a tag is specified indicating a specific data type stored across a first plurality of records in a first plurality of instances of a first field, the specific data type being associated with the number of distinct values ​​stored across the first plurality of records in the first plurality of instances of the first field; and The machine learning model is trained based on the training dataset. Receive a target time series dataset for a first time interval, the target time series dataset comprising a second plurality of records associated with corresponding plurality of consecutive second time intervals, each record comprising the same set of fields, the set of fields comprising a second plurality of instances of a second field and a third plurality of instances of a third field; and Applying the machine learning model to the target time series dataset to classify first data in a second plurality of instances stored across a second plurality of records in a second field as corresponding to a first data type, and classifying second data in a third plurality of instances stored across a second plurality of records in a third field as corresponding to a second data type, wherein applying the machine learning model to the target time series dataset includes: Determine (a) a first number of distinct values ​​stored in a second plurality of instances of a second field across a second plurality of records and (b) a second number of distinct values ​​stored in a third plurality of instances of a third field across a second plurality of records, wherein the second plurality of records includes a first record containing a second plurality of instances of a second field and a third plurality of instances of a third field; Based at least on the first number of distinct values ​​stored in the second plurality of instances of the second field across the second plurality of records, the first values ​​stored in the second plurality of instances of the second field of the first record are classified as corresponding to the first data type; and Based at least on the second number of different values ​​stored in the third field of the first record in the third field of the first record, the second values ​​are classified as corresponding to the second data type.

9. The method of claim 8, wherein applying the machine learning model to the target time series dataset further comprises: The third data stored in the fourth instance of the fourth field across the second or more records is classified as a third data type based at least on the number of distinct values ​​stored in the fourth instance of the fourth field across the second or more records.

10. The method of claim 9, wherein classifying the first data as corresponding to the first data type includes classifying the first data as name data associated with a device name. The second data is categorized into data types corresponding to the second data type, including classifying the first data as use case data describing the operating status of the device. The third data is categorized into categories corresponding to the third data type, including runtime data that includes measurements generated by the device.

11. The method of claim 9, wherein training the machine learning model to classify the first data into a category corresponding to the first data type comprises: The machine learning model is trained based on determining that multiple values ​​stored in a second plurality of instances of the second field do not change across the second plurality of records, and classifies the first data as corresponding to the first data type.

12. The method according to claim 9, further comprising: Based on classifying the first data into a first data type, the first data is stored according to the first storage configuration; as well as Based on classifying the second data into a second data type, the second data is stored according to a second storage configuration that is different from the first storage configuration.

13. One or more non-transitory machine-readable media storing instructions that, when executed by one or more processors, cause to perform operations including: Training a machine learning model to classify values ​​in time series data, the training including: Obtain the training dataset; each training dataset contains historical data and includes: A time-series dataset corresponding to a specific time interval, the time-series dataset comprising: a first plurality of instances of a first field of a first plurality of records corresponding to the corresponding first plurality of time intervals; and For each record, a tag is specified indicating a specific data type stored across a first plurality of records in a first plurality of instances of a first field, the specific data type being associated with the number of distinct values ​​stored across the first plurality of records in the first plurality of instances of the first field; and The machine learning model is trained based on the training dataset. Receive a target time series dataset for a first time interval, the target time series dataset comprising a second plurality of records associated with corresponding plurality of consecutive second time intervals, each record comprising the same set of fields, the set of fields comprising a second plurality of instances of a second field and a third plurality of instances of a third field; and Applying the machine learning model to the target time series dataset to classify first data in a second plurality of instances stored across a second plurality of records in a second field as corresponding to a first data type, and classifying second data in a third plurality of instances stored across a second plurality of records in a third field as corresponding to a second data type, wherein applying the machine learning model to the target time series dataset includes: Determine (a) a first number of distinct values ​​stored in a second plurality of instances of a second field across a second plurality of records and (b) a second number of distinct values ​​stored in a third plurality of instances of a third field across a second plurality of records, wherein the second plurality of records includes a first record containing a second plurality of instances of a second field and a third plurality of instances of a third field; Based at least on the first number of distinct values ​​stored in the second plurality of instances of the second field across the second plurality of records, the first values ​​stored in the second plurality of instances of the second field of the first record are classified as corresponding to the first data type; and Based at least on the second number of different values ​​stored in the third field of the first record in the third field of the first record, the second values ​​are classified as corresponding to the second data type.

14. The one or more non-transient machine-readable media according to claim 13, wherein applying the machine learning model to the target time series dataset further comprises: The third data stored in the fourth instance of the fourth field across the second or more records is classified as a third data type based at least on the number of distinct values ​​stored in the fourth instance of the fourth field across the second or more records.

15. One or more non-transitory machine-readable media according to claim 14, wherein classifying the first data as corresponding to the first data type includes classifying the first data as name data associated with a device name. The second data is categorized into data types corresponding to the second data type, including classifying the first data as use case data describing the operating status of the device. The third data is categorized into categories corresponding to the third data type, including runtime data that includes measurements generated by the device.

16. The one or more non-transitory machine-readable media of claim 13, wherein training the machine learning model to classify the first data into a first data type comprises: The machine learning model is trained based on determining that multiple values ​​stored in a second plurality of instances of the second field do not change across the second plurality of records, and classifies the first data as corresponding to the first data type.

17. One or more non-transitory machine-readable media according to claim 13, wherein the operation further comprises: Based on classifying the first data into a first data type, the first data is stored according to the first storage configuration; as well as Based on classifying the second data into a second data type, the second data is stored according to a second storage configuration that is different from the first storage configuration.

18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more hardware processors, cause to perform operations including: Receive a time series dataset for a first time interval, the time series dataset comprising multiple records associated with corresponding multiple consecutive second time intervals, each record comprising the same set of fields, the set of fields comprising a first plurality of instances of a first field and a second plurality of instances of a second field; Analyze a first plurality of instances of a first field across the plurality of records to determine a first number of values ​​stored in a first plurality of instances of a first field across the plurality of records; Based on a first quantity equal to or less than a first threshold: classify the first data stored in the first plurality of instances of the first field as corresponding to the first data type; Analyze a second plurality of instances of the second field across the plurality of records to determine a second number of values ​​stored in a second plurality of instances of the second field across the plurality of records; Based on the second quantity exceeding the first threshold: classify the second data stored in the second multiple instances of the second field as corresponding to the second data type; Based on classifying the first data into a first data type, the first data is stored according to the first storage configuration; as well as Based on classifying the second data into a second data type, the second data is stored according to a second storage configuration that is different from the first storage configuration.

19. One or more non-transitory computer-readable media according to claim 18, wherein the operation further comprises: Analyze a third plurality of instances of the third field across the plurality of records to determine a third number of distinct values ​​stored in a third plurality of instances of the third field across the plurality of records; as well as Based on a second threshold where the third quantity exceeds a first threshold: the third data stored in the third multiple instances of the third field is classified as corresponding to the third data type.

20. One or more non-transitory computer-readable media according to claim 19, wherein the first data type is name data associated with a device name. The second data type is use case data that describes the operating status of the device. The third data type includes runtime data of measurements generated by the device.

21. A system comprising: At least one device, including a hardware processor; The system is configured to perform the following operations: Training a machine learning model to classify values ​​in time series data, the training including: Obtain the training dataset; each training dataset contains historical data and includes: A time-series dataset corresponding to a specific time interval, the time-series dataset comprising: a first plurality of instances of a first field of a first plurality of records corresponding to the corresponding first plurality of time intervals; and For each record, a tag is specified indicating a specific data type stored across a first plurality of records in a first plurality of instances of a first field, the specific data type being associated with the number of distinct values ​​stored across the first plurality of records in the first plurality of instances of the first field; and The machine learning model is trained based on the training dataset. Receive a target time series dataset for a first time interval, the target time series dataset comprising a second plurality of records associated with corresponding plurality of consecutive second time intervals, each record comprising the same set of fields, the set of fields comprising a second plurality of instances of a second field and a third plurality of instances of a third field; and Applying the machine learning model to the target time series dataset to classify first data in a second plurality of instances stored across a second plurality of records in a second field as corresponding to a first data type, and classifying second data in a third plurality of instances stored across a second plurality of records in a third field as corresponding to a second data type, wherein applying the machine learning model to the target time series dataset includes: Determine (a) a first number of distinct values ​​stored in a second plurality of instances of a second field across a second plurality of records and (b) a second number of distinct values ​​stored in a third plurality of instances of a third field across a second plurality of records, wherein the second plurality of records includes a first record containing a second plurality of instances of a second field and a third plurality of instances of a third field; Based at least on the first number of distinct values ​​stored in the second plurality of instances of the second field across the second plurality of records, the first values ​​stored in the second plurality of instances of the second field of the first record are classified as corresponding to the first data type; and Based at least on the second number of different values ​​stored in the third field of the first record in the third field of the first record, the second values ​​are classified as corresponding to the second data type.

22. A system comprising: At least one device, including a hardware processor; The system is configured to perform the following operations: Receive a time series dataset for a first time interval, the time series dataset comprising multiple records associated with corresponding multiple consecutive second time intervals, each record comprising the same set of fields, the set of fields comprising a first plurality of instances of a first field and a second plurality of instances of a second field; Analyze a first plurality of instances of a first field across the plurality of records to determine a first number of values ​​stored in a first plurality of instances of a first field across the plurality of records; Based on a first quantity equal to or less than a first threshold: classify the first data stored in the first plurality of instances of the first field as corresponding to the first data type; Analyze a second plurality of instances of the second field across the plurality of records to determine a second number of values ​​stored in a second plurality of instances of the second field across the plurality of records; Based on the second quantity exceeding the first threshold: classify the second data stored in the second multiple instances of the second field as corresponding to the second data type; Based on classifying the first data into a first data type, the first data is stored according to the first storage configuration; as well as Based on classifying the second data into a second data type, the second data is stored according to a second storage configuration that is different from the first storage configuration.

Images