Interface for processing sensor data with hyperscale services

Abstract

A system for storing aggregated sensor data in a relational database, comprising: a storage facility; one or more processing units coupled to the memory; and one or more computer-readable storage media that store instructions which, when loaded into memory, cause one or more processing units to perform operations that include: by means of a data collection service (120, 312) periodically reading sensor data from a plurality of devices from a streaming service (114), wherein each device of the plurality of devices comprises one or more hardware sensors; Determining a set of attributes or metadata elements to be applied to the sensor data and annotating the sensor data by a tagging component to produce tagged data, wherein the annotation of the sensor data includes adding metadata to the sensor data or editing metadata of the sensor data, wherein the one or more hardware sensors are grouped into one or more indicator groups, and wherein the tagged data is stored as time series data in a data store; by an aggregation engine (356), aggregating data for a group by generating one or more aggregated values ​​from the tagged data stored in the data store; through an aggregation engine (356), writing the aggregated values ​​as aggregates in a semi-structured format back into data storage; and by a database writer (374), instantiating a relational database table, wherein the relational database table comprises for each respective hardware sensor in an indicator group a respective column in which aggregate values ​​for the respective hardware sensor are stored; and Calling a data read method to retrieve data for the aggregates and writing the aggregates to the relational database table.

Description

AREA

[0001] This disclosure relates generally to techniques for processing data using a hyperscale computing system. In a specific example, an interface to one or more hyperscale computing systems is created. BACKGROUND

[0002] As computing devices become smaller and more powerful, ever-larger amounts of data are available for analysis. For example, sensors (which may be integrated into smaller devices that are in turn connected to larger devices) that are connected to a network either wirelessly or via a wired connection are increasingly being integrated into devices or environments. These interconnected devices can be referred to as the Internet of Things (IoT). The amount of data generated by IoT devices can be enormous. Accordingly, there is a need for improvement in how IoT data is handled, including its collection and processing, so that it is available for applications. SUMMARY

[0003] This summary serves to present a selection of concepts in simplified form, which are explained in more detail below. This summary is neither intended to identify key features or essential characteristics of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] Techniques and solutions for processing data in conjunction with one or more hyperscale computing systems are provided. An interface is provided to translate calls from an application into a format used by a hyperscale computing system. These calls can involve reading data from or writing data to a hyperscale computing system. In certain examples, the data to be read or written is data from a variety of IoT devices, each IoT device having one or more hardware sensors. An interface can also be used to configure how the hyperscale computing system processes data, for example, determining how IoT data is stored or how aggregates are generated from IoT data.

[0005] One aspect provides a method for processing data in a hyperscale computing system. An application receives a request via an interface to access data stored in the hyperscale computing system, for example, to read data from the hyperscale computing system, write data to the hyperscale computing system, or modify configuration data stored in the hyperscale computing system. The interface translates the request into a format used by the hyperscale computing system to provide a formatted request. The formatted request is then transmitted to the hyperscale computing system.

[0006] The present disclosure also includes computer systems and physical, non-volatile, computer-readable storage media configured to perform a procedure described above or containing instructions for performing such a procedure. As described herein, a variety of further features and advantages can be incorporated into the technologies as desired. BRIEF DESCRIPTION OF THE DRAWINGS • Fig. Figure 1 is a diagram of a computing environment in which the disclosed technologies can be implemented, wherein the computing environment facilitates the writing of data from or generated by IoT devices to an analysis platform. Fig. 2A is a JSON listing of a sample message from an IoT device. Fig. 2B is a JSON sample message from an IoT device, enriched with additional metadata elements. Fig. Figure 3 is a diagram of a computing environment in which the disclosed technologies can be implemented, wherein the computing environment facilitates the writing of data from IoT devices or data generated by them, stored in a hyperscale computing system, to an analysis platform. Fig. 4 is a JSON listing of a sample message indicating that a new aggregate is available for processing and storage in an analytics platform. Fig. Figure 5 shows various table schemas that can be used to store data from or generated by IoT devices. Fig. Figure 6 is a diagram of a computing environment in which the disclosed technologies can be implemented, wherein the computing environment facilitates the storage of application master data on an analytics platform where it can be combined with time series data generated by IoT devices. Fig. 7A is an example of a JSON listing of a message for deleting master data. Fig. Figures 7B-7D show an example of a JSON listing of a message for adding master data. Fig. 7E-7H show an example of a JSON listing of a message for changing master data. Fig. Figure 8A shows an excerpt from a source master data schema. Fig. Figure 8B shows a master data schema that can be used for data analysis and is based on the source master data schema. Fig. 8A can be assigned. Fig. Figure 9 is a flowchart of an example procedure for processing requests to access data in a hyperscale computing system using an interface. Fig. Figure 10 is a diagram of an example computing system in which some of the described embodiments can be implemented. Fig. 11 is an example of a cloud computing environment that can be used in conjunction with the technologies described here. DETAILED DESCRIPTION Example 1 - Overview

[0007] As computing devices become smaller and more powerful, ever-larger amounts of data are available for analysis. For example, sensors (which may be integrated into smaller devices that are in turn connected to larger devices) that are connected to a network either wirelessly or via a wired connection are increasingly being integrated into devices or environments. These interconnected devices can be referred to as the Internet of Things (IoT). The amount of data generated by IoT devices can be enormous. Accordingly, there is a need for improvement in how IoT data is handled, including its collection and processing, so that it is available for applications.

[0008] In many applications, IoT data from IoT devices (sometimes called edge devices) is combined with other data that provides context or otherwise enriches the IoT data to make it usable. IoT data can contain relatively little descriptive information, which can be beneficial for several reasons. Transmitting reduced amounts of information can conserve network and processing resources on an IoT device and also lower power consumption. Furthermore, a limited amount of descriptive information in IoT data can be useful for enabling a change in the end use of the IoT data without requiring changes to the IoT device or the format in which the IoT data is transmitted.

[0009] Typically, IoT data is not in a format suitable for its eventual processing. This means that, in addition to lacking descriptive information, IoT data is often in a semi-structured format that may not be well-suited for analysis and visualization. For example, many applications that query and process data (including various analytics applications) may be configured to use structured data, such as data stored in a relational database system. To provide context for IoT data and facilitate its processing and analysis, it can be beneficial to combine the IoT data with information stored in a structured format, such as master data or other attributes related to the IoT data / devices, which may be stored in a relational database format.Storing IoT data and master data (or other descriptive data) in a common format, such as a relational database system, can help to combine or otherwise process data from these different sources, for example by performing JOIN operations (e.g., in SQL or another query language).

[0010] Many applications therefore require two processes to enable analytical queries or other data processing. First, IoT data from IoT devices must be captured and converted into a format that allows it to be combined with master data or other structured data. Second, such master data or other structured data must be obtained. These processes can be complex for a number of reasons, including when a system enabling analytical applications for IoT data can be used with a variety of IoT devices / IoT platforms, by a number of different entities (e.g.,(clients in a multi-tenant database or other multi-tenant application architecture) and may be associated with master data or other descriptive data that may originate from a number of different data sources, which may have different data models or schemas, and where the original data of the schemas are not optimized for analytical applications.

[0011] This disclosure provides technologies that can be used to solve one or more of the problems mentioned above. For simplicity, much of the following disclosure refers to “cloud-based systems.” Cloud-based systems may also be called hyperscale computing systems, and these terms are used interchangeably in this disclosure. IoT data may initially be captured by a cloud-based system, processed IoT data may be stored in a cloud-based system, and master data may originate from cloud-based applications and be stored in the cloud-based system along with the IoT data. The cloud-based systems may be identical or different, depending on the implementation.

[0012] However, it should be noted that the disclosed technologies are not limited to cloud-based systems unless this is clear from the context of the discussion. For example, the disclosed technologies can be used with remote computing systems, which are not typically considered "cloud-based." Furthermore, one or more systems may be non-cloud-based / non-remote systems, such as those running on a computing system or device maintained by an entity.

[0013] Cloud-based systems are often provided by a company that is distinct from the company whose users access the cloud-based system. However, the technologies revealed can be applied to computing systems connected to the same company. Similarly, cloud-based resources can be physically farther away than in other types of network environments. That is, in a network environment, a computing device might be considered "remote" from another computing device if they reside on separate machines or if they do not share certain computing resources (e.g., a file system, a namespace, or storage space). On the other hand, cloud-based systems can encompass machines located many miles away from the devices accessing the cloud-based system's resources.A remote computing device or system, as described above, therefore refers to physically separate computing devices that usually communicate over a network, although no specific degree of "distance" is required.

[0014] Accessing remote resources can also be desirable because it can be more cost-effective than maintaining a local system. For example, if a cloud provider handles infrastructure issues such as providing suitable hardware, users can operate more flexibly, scaling their use of cloud services up or down as needed. Having a provider maintain and configure the software can reduce costs for businesses, as they require less skilled personnel. Obtaining software as a service from a provider can improve performance and availability, as the provider may be better equipped to update and configure the software.

[0015] As previously mentioned, IoT data received from an IoT device typically contains relatively limited identifying information or metadata and consists mainly of measurements or similar information recorded by sensors connected to the IoT device. In one aspect, the present disclosure provides a capture process in which each measurement from an IoT device is annotated (or tagged) with a selected set of attributes (e.g., a type of metadata, which may also be called a metadata element or property). These attributes may be or may contain attributes selected from a set of master data. Annotating attributes can facilitate the storage and processing of the IoT data by, for example, specifying to an aggregation component how aggregations (where an aggregation may also be called a composite or composite value) should be calculated (e.g.,An annotation specifies an aggregation frequency, an aggregation granularity, values ​​calculated during aggregation, or a storage component, indicating where / how individual IoT data messages or aggregates should be stored. Such annotations can be particularly useful when a system processes IoT data from multiple clients (e.g., tenants) to ensure that the IoT data for one tenant is stored in a way that is at least logically separate from the data of other tenants.

[0016] Annotations can be defined for specific types or sets of IoT devices, specific tenants, and so on. In at least some cases, attributes used for annotations, or values ​​for these annotations, can be retrieved dynamically. When an IoT message is received, APIs can be used to obtain information about which attributes should be used to annotate specific messages. Alternatively, the APIs can be used to determine specific values ​​for certain attributes to be used to annotate a particular IoT data message received (directly or indirectly) from an IoT device. Since retrieving the attributes / values ​​can be time-consuming and resource-intensive, an annotation cache can be maintained. In some cases, annotation information can be dynamic / configurable.When a configuration changes, a notification can be sent to an annotation component, which can optionally invalidate cached attributes or attribute values.

[0017] As explained above, in at least some implementations, master data or other attributes that can be used with IoT data can be received from other applications or data sources. This disclosure provides a master data capture component / pipeline that can receive messages from applications, where the messages contain at least some master data / attributes or provide information that can be used to retrieve master data / attributes or other information that can be used in conjunction with data analytics applications. In certain cases, a notification is analyzed to determine information necessary to process a request relating to master data / data attributes, including retrieving master data / attributes from specific applications or data sources (which may include an application that generated the message).

[0018] In some cases, master data / attributes can originate from multiple sources, where the master data may be maintained in different schemas (e.g., tables may contain different attributes, attributes may have different names or data types), or a source schema may not be optimized for analysis applications, especially for applications involving IoT data. The disclosed technologies provide an analysis data schema that can be used as a common schema for master data / attributes from source schemas of multiple sources and is better optimized for analysis. In at least some cases, data in the analysis / common schema is denormalized with respect to at least some of the corresponding tables (or other sources) in the original data model.This denormalization / consolidation can lead to better performance, for example by reducing query processing time and resource usage through a reduction in table JOIN operations.

[0019] Various IoT data sources, such as cloud storage / hyperscale computing systems, can be connected using different protocols, methods, interfaces, and so on. In one aspect, this disclosure provides an interface that can be called by components involved in retrieving and processing IoT data. This allows the underlying details of a cloud provider's or other IoT data service's operation to be abstracted from the application. One advantage of this technique is that the same pipeline can be used with a variety of cloud providers / IoT data sources, enabling, for example, different clients using different platforms to use the same pipeline, or facilitating a client's ability to switch between different platforms.The interface can include methods for writing data, such as the tagged or annotated data described above, to the IoT data service / platform, or for reading data from the service / platform, such as writing aggregates generated by the service / platform to a relational database or other structured data storage / retrieval system. An interface method can also allow users to configure other aspects of the service / platform, such as how data is stored or how aggregates are calculated.

[0020] Especially given the potentially large volume of IoT data, it can be crucial to develop systems that efficiently process it into a format more easily processed by end-user applications (e.g., for IoT data analytics). Efficient storage of such IoT data is also essential. One aspect of this involves defining tables for specific groups of attributes. These attributes can be those directly related to IoT data or those related to master data / semantic attributes used to provide context for the IoT data. For example, sensors or indicators from IoT devices can be grouped into indicator groups. A separate table can be defined for each indicator group.

[0021] In particular, when aggregates generated from IoT data are calculated and optionally stored based on indicator groups, storing data based on indicator groups can enable efficient data processing. For example, a thread performing operations for a database writer can perform read and write operations on aggregates for one indicator group in a table for that indicator group without interfering with other threads that might be performing operations on other indicator groups. This can reduce problems such as lock conflicts and delays, and increase concurrency.

[0022] In addition to storing IoT data aggregates in tables organized by indicator groups, it can be beneficial to structure a table so that each indicator group has columns for each indicator (sensor, IoT data source). This table formatting allows data types to be maintained for each indicator, unlike a table schema that has one column for an indicator identifier and one column for a value for that indicator. In such a schema, the "value" column would typically reduce all data types (e.g., integers, floating-point numbers, dates, Boolean values) to a common data type (e.g., a string), which can be critical for downstream data processing applications. Furthermore, providing columns for each indicator, especially if the tables are in a columnar format (i.e.,managed in a columnar memory (as opposed to a row memory), enabling data compression and facilitating the modification of indicator groups.

[0023] Another advantage of the table schema described above is that it allows for the easy creation of new indicator groups or the modification of existing ones. In the case of a new indicator group, a table can be easily created using rules that define columns for specific attributes of the IoT data, as well as columns for aggregated values ​​for each indicator. When an indicator is added to an indicator group, the table can be modified to include a column for the indicator. Optionally, aggregated values ​​for existing rows can be calculated for the added indicator / column. When an indicator is removed from an indicator group, the corresponding column can be removed from the table. In at least some cases, the column is not deleted or not immediately removed, but rather excluded from the results for the indicator group (e.g.,(based on a definition of the indicator group that is linked to the table or otherwise used to define the data to be retrieved from the table). Moving an indicator between indicator groups can be done by combining the operations described above for adding / removing indicators.

[0024] Particularly given some IoT device operating environments, it's possible that some indicators may not transmit their data in real time or within the timeframe in which such data is typically expected. Typical IoT data pipelines are unable to retrospectively process delayed IoT data, or even IoT data received after a certain period. The disclosed technologies enable the processing of delayed data. An aggregation process can detect when new IoT data becomes available and generate a new or updated aggregation, which can then be made available for further processing.

[0025] A process that writes data to a database table can determine whether the data to be written is new or updated and can add or update rows accordingly. In some cases, non-aggregated data may be deleted from storage (e.g., cloud storage) before aggregated data is deleted. Certain technologies store information along with aggregated values ​​(e.g., data point counters), which allows updated aggregated values ​​to be calculated (e.g., using weights based on an updated count) even if data for the individual data points is no longer available. Example 2 - Example of annotating IoT device messages

[0026] Fig. Figure 1 shows an example of a computing environment 100 that can be used to acquire and process IoT data from one or more IoT devices 102. The IoT devices 102 can be associated with a specific environment, such as a particular machine or physical location 106. IoT devices 102 can be grouped into one or more groups 108. Groups of IoT devices 102, of specific sensors of IoT devices, or combinations thereof can be called indicator groups.

[0027] IoT devices 102 can be grouped based on one or more criteria. For example, all IoT devices 102 at a specific location 106 (e.g., a production line) or those serving a specific purpose can be grouped. Grouping can also be done based on one or more types of measurements provided by an IoT device 102 or a specific component (e.g., a sensor) of it. For example, all or some of the IoT devices 102 containing temperature sensors can be grouped together, while pressure sensors can be collected in another group.

[0028] IoT devices (102) are generally network-enabled devices. Although "Internet of Things" devices can refer to devices with a wide variety of characteristics, in at least some cases, they can refer to devices with embedded computing systems, with hardware or software for special purposes (e.g., as opposed to general-purpose computing devices), or a device that is not traditionally associated with computers or network connectivity.

[0029] For example, refrigerators have been around for quite some time, and they typically had little to no computing power (e.g., controlling lighting and cooling components, providing a display, receiving user input), capabilities that would not usually be considered network connectivity. However, refrigerators are now being manufactured with embedded computing devices that include network connectivity, such as Wi-Fi, Bluetooth, ZigBee, Z-Wave, cellular networks, near-field communication, Sigfox, Neul, or LoRaWAN. By adding network connectivity and potentially additional computing power or functionality (though usually still limited compared to more general computing devices like PCs, tablets, and smartphones), the refrigerator can be transformed into an IoT device.

[0030] Devices for the Internet of Things, or devices 102 that are useful in other aspects of this disclosure, typically include one or more hardware sensors. That is, devices for the Internet of Things can collect information about their environment or their use and transmit this information over a network. In some cases, a particular IoT device 102 (device 102a, as shown) may have multiple sensors 110, including sensors of different types. In such cases, groupings (e.g., indicator groups) can be performed at the granularity level of individual sensors 110 of IoT devices 102.

[0031] The sensors used by Device 102 may include position sensors that determine the relative or absolute position of an object. Position sensors may include those typically contained in an inertial measurement unit (IMU), including accelerometers, gyroscopes, magnetometers, and combinations thereof. For example, a gate, door, or turnstile may be equipped with one or more accelerometers, with a defined degree or range of movement indicating that the access control device has been activated and an access event has occurred. Position sensors may also include sensors that determine an object's geospatial location, for example, using a global navigation satellite system (GNSS, such as the Global Positioning System or GPS). The geospatial position may be determined by other types of position sensors, such as…Wireless transceivers can be determined. That is, a receiver component can use techniques such as triangulation and signal strength measurements to determine a position.

[0032] The sensors used by the devices 102 may also include radiation sensors to detect, for example, when a person or object has entered a proximity to the sensor, such as an area controlled or monitored by the access control devices. Suitable radiation sensors include infrared and visible light sensors, as well as ultrasonic sensors. Radiation sensors may also include cameras, such as a still camera or a video camera.

[0033] The 102 devices can be used for device monitoring. When used for device monitoring, the 102 devices can monitor the status of a device or system (equipment) on which the device is installed. Device monitoring can be used, for example, to determine how often the device or a component thereof has been activated. This information can be used to determine the operational or maintenance status of the device, such as whether the device is due for scheduled or preventive maintenance.

[0034] Parameters such as operating temperatures and pressures can be monitored to determine whether the device is operating within normal parameters, requires repair, needs adjustment to operating conditions, or requires preventive or scheduled maintenance. An unusually high temperature associated with the device may indicate that a specific component or the device itself is prone to failure and should be replaced. In addition to potentially reducing repair or replacement costs (e.g., because a device is easier to service before it fails), device monitoring with Device 102 can reduce equipment downtime and mitigate health and safety concerns.

[0035] In at least some implementations, data is sent from the IoT devices 102 to an IoT or cloud service 112. The IoT or cloud service 112 is typically configured to receive and store data from IoT devices. The IoT or cloud service 112 can send IoT data to a streaming service 114. The streaming service 114 can have a publisher-subscriber relationship with the IoT or cloud service 112. The streaming service 114 can include software such as KAFKA, available from the Apache Software Foundation (Forest Hill, MD). Data can be stored in the streaming service 114 in one or more containers 116 (which can be configured as containers for storing time-series data). Typically, the containers 116 are associated with a specific topic, and data from specific IoT devices 102 (or their sensors) is routed to the appropriate container.

[0036] Data in the streaming service 114 can be regularly read by a data collection service 120, for example, by a consumer component 124 of the data collection service. IoT data received by the consumer component 124 can be annotated by a tagging component 128. The tagging component 128 can add or modify data, such as metadata, for each element (e.g., a message or other discrete reading or measurement provided by an IoT device 102) of the data sent by an IoT device. The annotations applied by the tagging component 128 can correspond to attributes, such as attributes from a master data schema of a relational database system.The tags can be used in further processing of the IoT data, for example to determine where / how data is stored, how long data is retained, how data is aggregated, where aggregated data is stored, how long aggregates are retained, into which table data elements from the IoT data are written, and they can provide values ​​that are contained in a table (or other structured data format) into which data elements of the IoT data are written.

[0037] For example, various annotations can be applied to IoT data to indicate how an IoT device 102 relates to another device, such as a device into which the IoT device is integrated or with which it is otherwise connected, or to indicate how multiple IoT devices relate to each other. As described, individual IoT devices 102 or their sensors can correspond to an indicator that is part of an indicator group. Thus, when IoT data is analyzed by the tagging component 128, the IoT data can be annotated with the indicator group to which the IoT data is associated.The indicator group can be determined, for example, by comparing an identifier for the IoT device contained in the IoT data with a directory or mapping (which may be in the form of a table, such as a relational database table) that assigns IoT device identifiers to one or more indicator groups.

[0038] An IoT device 102 can be associated with a specific piece of equipment, such as a pump, a robot component, etc. The specific piece of equipment to which an IoT device 102 is associated can be specified by an equipment ID attribute, which can represent another metadata element used to tag IoT data. A specific piece of equipment can be associated with a particular type, which can be specified by a model ID attribute used to annotate IoT data. A set of models for a particular type of equipment may, in turn, share common components or functions, so a template ID attribute can be used to specify a set of one or more attributes relevant to a particular model that falls under the template.In other words, attributes can be arranged in hierarchical groups, for example in the order from general to specific of the template ID, model ID, equipment ID and indicator group (along with an identifier of the actual IoT device 102 / sensor that transmitted the IoT data).

[0039] Other types of metadata that can be added to IoT data by tagging component 128 include a tenant identifier if the data is managed in a multi-tenant system, or otherwise an identifier for a specific client, system, or application that is to receive the data. Other metadata elements can be used to tag IoT data with information such as aggregation or data processing policies, including whether data should be encrypted or protected (e.g., confidential or privileged).

[0040] Fig. 2A shows an example of a raw, untagged IoT data message 200 from an IoT device 102. Fig. 2B shows a tagged IoT data message 250, which is derived from IoT data message 200. Fig. 2A corresponds to this after it has been processed by the tagging component 128. It should be noted that the untagged IoT message from Fig. 2A can represent the IoT data after it has undergone processing / formatting / annotation by upstream processes / components. For example, the IoT message from Fig. 2A is represented in JSON format (JA-VASCRIPT OBJECT NOTATION), while the data from an actual IoT device may be in a different format (e.g., string, XML, character, CSV). In some cases, the IoT device can natively send 102 messages in JSON format, but the content can optionally be extended or reformatted (e.g., key labels changed to key / value pairs).

[0041] It is evident that the tagged IoT data messages in Fig. 2B removes some data elements (e.g., thingld, thingType) from the untagged message and adds other elements, including tags 254, an "identifier" 258, and a "structureid" 262. While the "tenant" metadata element is present in both Fig. 2A as well as in Fig. If 2B is present, in other cases the client identifier can be determined during the processing of the untagged IoT data message 200 and added during the tagging / annotation process that generates message 250.

[0042] To execute the tagging function, the tagging component 128 determines for a given IoT data message which set of attributes (metadata elements) to apply and which values ​​to use with that specific message. Since the set of attributes can potentially be applied to IoT data messages from many IoT devices 102, the set of attributes to be used with a particular IoT device is typically managed by the tagging component 128. With reference to Fig. 1. Data elements in the untagged IoT data message, such as thingD or thingType, can be compared with a property set mapping 132 to determine a property set 136. In other cases, the untagged IoT message 200 contains an identifier for a property set 136 or a set of properties to be used for tagging, with the values ​​for the properties being determined during an annotation process.

[0043] Although a set of properties intended for tagging untagged messages from IoT devices 102 may be consistent for at least a subset of the devices, the actual values ​​to be used with the tags may vary more significantly, including in some cases where different IoT devices, even of the same type, are associated with different values ​​for a particular metadata element. When an untagged message is processed by the tagging component 128, the tagging component can determine which properties to use for tagging the message (e.g., from the property mapping 132 / property sets 136). The values ​​to be used with the properties may be obtained from one or more external sources, for example, using REST APIs 140.

[0044] In some cases, REST APIs may contain methods for various external sources or different property sets. In other cases, REST APIs may be more general. The arguments provided in a call to a REST API may depend on the API implementation. If the REST API is more specific, an identifier for the respective IoT device (or a group to which the IoT device belongs) may be provided as an argument. If the REST API is more general, API calls may also include identifiers for specific properties whose values ​​are to be retrieved, along with an identifier for the relevant IoT device or IoT device group.

[0045] Using REST APIs 140 can be time-consuming and resource-intensive. Therefore, the ingestion service 120 can include a cache 144 for storing property / value information. When the tagging component 128 parses an untagged message, after determining which properties to use for tagging the message, it can consult the cache 144 to see if the required values ​​are stored there. If so, the tagging component 128 can retrieve the values ​​from the cache 144 and tag the message. The cache 144 can optionally be updated to log access to the values ​​and, optionally, information such as the access time. Storing information about retrieving values ​​from the cache 144 can be useful for certain cache management strategies (e.g., "Least Recently Used," "Least Frequently Used").

[0046] If the required values ​​are not present in cache 144, the tagging component 128 can request the values ​​via the REST APIs 140. Once the REST APIs 140 receive the values, they can be stored in cache 144 and returned to the tagging component 128.

[0047] After tagging, the messages can be sent from the capture service 120 to a hyperscale computing system 150, which can be a cloud-based data storage and processing service (e.g., Amazon Web Services, Microsoft Azure, Google Cloud Platform), for example, by a writer 148. As explained in more detail in Example 3, the hyperscale computing system 150 can perform operations such as storing raw, tagged messages from the capture service 120, performing aggregations, storing aggregates, and notifying other components of a data processing pipeline that new aggregates are available (e.g., notifying a component of a relational database system that aggregates are available to write to the database system).Although described as being sent to a hyperscale computing system 150, in other embodiments the tagged messages can be sent to a different type of system for storage or further processing.

[0048] As described in more detail in the following examples, the hyperscale computing system can generate 150 aggregates, which are written as time series data 162 to a data store 158 of an analysis platform 154. The time series data 162 can be converted into a structured format, such as a relational database table, and stored there. The data store 158 can also store master data 166, which can be combined with the time series data 162 for analysis. The master data 166 can correspond to master data 174 of one or more applications 170. Later examples describe how the master data 174 can be retrieved from the applications 170 and optionally converted for use with the analysis platform 154.

[0049] As further described, a client 180 can access the analysis platform 154 to, for example, perform data analyses and configure various aspects of the computer architecture 100. For example, the client 180 can issue commands to change how data is processed by the acquisition service 120 or stored or processed in the hyperscale computing system 150. The client 180 can make changes to the master data 174, which in turn can be passed on to the master data 166. Example 3 - Example of the interaction of an IoT data processing pipeline with a hyperscale computing system

[0050] Fig. Figure 3 illustrates a computing architecture 300 that can be used to migrate IoT data into a database system. The computing architecture 300 includes an IoT data platform 308 that sends data received from IoT devices 304 to a collection service 312. The collection service 312 can be part of an analytics service 302, which facilitates the analysis of IoT data, or otherwise interact with it.

[0051] The IoT data platform 308 can be used by the data streaming service 114. Fig. 1, while the capture service 312 can be the capture service 120. For clarity, the capture service 312 is shown only with a tagger 314 (e.g., the tagging component 128) and tag rules 316 (which, for example, include one or more of the property set mappings 132, property sets 136, the cache 144, and the REST APIs 140). Fig. 1 can include).

[0052] The data collection service 312, which can also be referred to as an IoT data consumer, interacts with a hyperscale computing system 320, which is the hyperscale computing system 150. Fig. 1. As described in Example 2, in other implementations, components of the Hyperscale Computing System 320 may be contained within a non-hyperscale system. Furthermore, the functionality provided by the Hyperscale Computing System 320 may not be separate from a system implementing a capture pipeline (which includes the Capture Service 312 and other components described below), but rather integrated into a computing system that implements the capture pipeline.

[0053] In some cases, the 312 ingestion service interacts with a fixed API, which in turn interacts with the 320 hyperscale computing system. However, it can be advantageous to allow the ingestion pipeline to operate flexibly with multiple 320 hyperscale computing systems (or computing systems with equivalent functionality). Accordingly, the 312 ingestion service can interact with one or more 320 hyperscale computing systems through a hyperscale interface 324 (e.g., by making calls to them), where the hyperscale interface handles a call and sends a formatted request to a 320 hyperscale computing system. The hyperscale interface 324 provides methods for performing various operations on a 320 hyperscale computing system, such as writing data to the hyperscale service, reading data from the hyperscale service, and determining whether new data (e.g.,Aggregates) are available in the Hyperscale service, or to configure the Hyperscale service.

[0054] The Hyperscale Interface 324 contains implementations of the provided methods for various Hyperscale Services 320, which can be used by the Ingestion Service 312 or other components of an ingestion pipeline. For example, the Ingestion Service 312 might initially be configured to work with Amazon Web Services as a Hyperscale Computing System 320, so the Hyperscale Interface 324 implements write, read, data validation, and configuration methods for Amazon Web Services. Later, it might be desirable to also process data with Microsoft Azure, so the Hyperscale Interface 324 can be updated to implement its methods for that platform.However, since the Capture Service 312 and other components of the capture pipeline access the Hyperscale Interface 324, they may not need to be changed (or only to a lesser extent) to work with a new Hyperscale Computing System 320.

[0055] Data sent by the ingestion service 312 to the hyperscale computing system 320, for example via the hyperscale interface 324, can first be processed by a data loader 328 (which can also be called a data ingestion component or streaming data service). The data loader 328 can perform actions to store data, such as tagged IoT messages 332, in the data store 336 of the hyperscale computing system 320. Actions that the data loader 328 can perform include batch processing (for example, waiting until a threshold number of messages is reached and then writing the batched messages to the data store 336), compression, and encryption. In a specific example, such as when AMAZON WEB SERVICES is used as the hyperscale computing system 320, the data loader could be the application FIREHOSE.

[0056] The data loader 328 can access a rule set 340. The rule set 340 can store configuration information about how different types of IoT messages 332 should be processed, such as the identification of a tenant container 344 (represented as 344a, 344b) in which data is to be stored, and optionally a subcontainer 348 (represented as 348a, 348b) in which data is to be stored (e.g., a storage path, such as a file path). In some cases, subcontainers 348 can be used for specific groups of devices or sensors (e.g., indicator groups), with the groupings being formed based on criteria such as sensors with a common purpose, a common sensor type (e.g., pressure data, temperature data), belonging to the same facility or facility category, or having a common location.Data, such as the marked messages 332, can be stored in time series files 352.

[0057] Although time-series files 352 can be organized in various ways, the data store 336 in a specific example uses a storage or partitioning scheme where messages 332 (IoT data) are partitioned by client, year, month, and day. Data files representing hourly data are stored in the path (e.g., folder) for each day. Optionally, the data files 352 can be further partitioned by another subcontainer 348, for example, a category as described above.

[0058] The data files 352 are processed by an aggregation engine 356. The aggregation engine 356 can periodically determine when new data files 352 are available for processing. In some cases, the aggregation engine 356 can be configured to determine whether data files 352 are available according to a specific schedule, such as hourly. However, it may be useful to check for data files 352 at other times or to check data files on demand (e.g., at the request of an end user or a software process). As explained below, in some implementations, the acquisition pipeline and the hyperscale computing system 320 can be configured to process late-arriving data (e.g., data sent late due to a connectivity issue with a particular sensor).By checking data files 352 at shorter intervals or on request, delayed data can be processed more quickly.

[0059] The aggregation engine 356 generates one or more aggregated values ​​from the tagged, individual data measurements stored in data store 336. Specifically, the aggregation engine 356 can generate aggregated values ​​over the period represented by a specific time-series data file 352 being processed—in this example, hours. Aggregated (or composite) values ​​generated by the aggregation engine 356 can include values ​​such as count, sum, minimum, maximum, average, median, standard deviation, and so on. The specific aggregated values ​​generated can be fixed, representing a default set, or customized, for example, in response to requests received through the hyperscale interface 324.

[0060] It can be advantageous to store information that allows for the revision of aggregates, even if the base files 352 have been deleted or are otherwise unavailable. For example, by storing count information (i.e., the number of data points in an aggregate), aggregate values ​​such as the average can be updated by weighting the new measurements accordingly.

[0061] Aggregated values ​​can be written back to data storage 336 by the aggregation engine 356 as aggregates 360. In some cases, the aggregates 360 can be stored in the same schema used for the time series files 352. In other cases, the aggregates 360 can be stored in a different schema. In particular, it may be desirable to aggregate data for specific indicator groups. For example, if the tagged messages 332 are stored by client / year / month / day, the aggregates 360 can be stored by client / indicator group / year / month / day or client / year / month / day / indicator group.

[0062] The aggregation engine 356 can contain a rule set 364. The rule set 364 can define various parameters used for processing different types of data files 352. For example, different clients 344 can have different aggregate values ​​calculated or aggregates generated with different frequencies (e.g., every half hour or every six hours instead of the default hourly aggregations). The rule set 364 can also define whether data in a data file 352 is further partitioned for aggregation (e.g., generating overall aggregate values ​​for a file in addition to or instead of generating aggregate values ​​based on indicator groups or other grouping criteria).

[0063] When a new aggregate 360 ​​is created, the aggregation engine 356 can send a notification to a queue service 368, which can manage one or more queues 370. In some cases, the queue 370 is a first-in, first-out (FIFO) queue. However, other types of queues, including priority queues, can be used if desired.

[0064] A database writer 374 in the ingestion pipeline can periodically check the queue 370 to determine if new aggregates 360 are available for processing. In some cases, multiple database writers 374 are available to write aggregates 360 to a database 386 of the analytics service 302. The database writer 374 can call a "Check for New Data" method provided by the hyperscale interface 324. When a new aggregate 360 ​​is available, information about the aggregate can be removed from the queue 370 and sent to the database writer 374. The information about the aggregate 360 ​​can include an identifier for the aggregate and a location from which the aggregate can be retrieved.

[0065] The database writer 374 can retrieve data for the aggregate 360 ​​by calling a "Read Data" method provided by the hyperscale interface 324. The database writer 374 can then write the data to the database 386 into one or more database tables 390 with one or more schemas 388, as described below. Each database writer 374 can contain a job (or task / item) queue 376 into which a task to write an aggregate can be queued after receiving the task from the queue 370. Each database writer 374 can contain a log 378, which is used to persistently record jobs associated with the database writer 374, for example, in case a job fails or the capture service 312 experiences an error. In other cases, a common log 378 can be used for all database writers 374.

[0066] In addition to the time series data associated with the aggregates 360, the tables 390 of the database 386 can also contain master data. Master data can be retrieved from one or more applications 391 by a master data service 392, and then processed by a master data consumer 394 to format the master data and write it to the database 386, as described below.

[0067] In some embodiments, the tables 390 can be arranged in star schemas (or snowflake schemas or similar schemas) 396, wherein a star schema contains a fact table 398a corresponding to the time series data and dimension tables 398b containing master data. In some cases, several dimension tables 398b can be combined (e.g., denormalized) into a single table to facilitate data processing (e.g., for OLAP queries or queries that otherwise perform analyses of the time series data). Example 4 - Example of an aggregated notification message

[0068] Fig. Figure 4 is an example of a message 400 that can be retrieved from queue 370 by database writer 374. Message 400 is a message in JSON format, using AMAZON WEB SERVICES as the hyperscale computing system 320. Message 400 contains a timestamp 404, indicating when the event was added to queue 370, which is typically at roughly the same time as an aggregation 360 is completed. Message 400 also includes a file location 412, shown with a file path 414 containing date information 416 and an identifier 418 of an indicator group associated with the aggregation available at that location.

[0069] Message 400 can contain additional information used to retrieve or process aggregates. For example, an account identifier 422 can be used to associate an aggregate with a specific user or process, among other things so that the aggregate can be written to the appropriate tenant store in a multi-tenant database system. Example 5 - Example of writing aggregates to a relational database

[0070] With renewed reference to Fig. 3. If available, a database writer 374 can consult the hyperscale computing system 320 to determine if new aggregates 360 are available for processing (for example, by calling a "check for aggregates" procedure of the hyperscale interface 324). If a new aggregate 360 ​​is available, the corresponding queue item 370 (aggregate notification) is removed from the queue and sent to the requesting database writer 374. In some implementations, the hyperscale computing system 320 maintains a log of items removed from the queue, at least for a certain period, to facilitate recovery in case of an error or system failure, such as a network failure while sending the aggregate notification to the database writer 374.

[0071] When database writer 374 receives an aggregate notification, the notification is added to its job queue 376. The notification, or an overall status of job queue 376, can be stored as entries in log 378. As described in Example 3, in some cases log 378 stores records of jobs / job status for all database writers 374. In other cases, separate logs 378 can be maintained for each database writer 374. In the event of a system failure or error, logs 378 can be read to assign incomplete jobs (retrieving aggregates and writing to tables in a database system) to database writers 374.

[0072] After a job is added to its job queue 376 and the job is stored in log 378, the database writer 374 retrieves the aggregate 360 ​​corresponding to the job, for example, by requesting the data from the hyperscale computing system 320 (e.g., using the hyperscale interface 324). The database writer 374 then proceeds to write the data for the aggregate 360 ​​to database 386. In general, the database writer 374's writing of the aggregate 360 ​​to database 386 transforms the IoT data (time series), now in aggregated form, from a semi-structured format (e.g., XML, JSON, CSV) into a structured format in the form of data stored in one or more relational database tables 390. Example 7 - Example database schemas for storing IoT data

[0073] The present disclosure provides a schema definition procedure that flexibly allows new tables to be added when new types of aggregates are required in a database (e.g., database 386 from Fig. 3) to be stored, and to change table formats when changes are received to the aggregate definition (e.g., for an indicator group). As described below, table schemas can be defined based on indicator groups. Because aggregates can be defined based on indicator groups, the definition of a table schema can correlate with the definition used for an aggregate that, according to the table schema, is to be stored in a table.

[0074] Correlating table schema definitions with aggregate definitions can offer several advantages. One advantage is that a table schema can be more easily adapted when a change is made to an aggregate definition (e.g., an indicator group associated with the definition, such as adding or removing sensors from an indicator group). Because table definitions are consistent with aggregate definitions, new tables can be created more easily when new aggregates (e.g., for new indicator groups) are created.

[0075] Another advantage of using aggregate definitions as the basis for table schemas is that multiple database writers can more easily work concurrently. That is, database writers writing different aggregates typically write to different tables, thus reducing the likelihood of one database writer having to wait for another to release table locks. Since aggregates are usually created on a schedule, typically one aggregate for a specific set of IoT devices (or individual sensors within them) is written to the database before the next aggregate for the same set of IoT devices is processed.

[0076] Fig. Figure 5 shows two example table schemas, 500 and 550, that can be used to store aggregate data. Both table schemas, 500 and 550, contain columns 510 and 512, which provide time information for a specific row in the table. As shown, columns 510 and 512 represent start and stop timestamps, respectively, for sensor values ​​in the aggregate. However, an aggregated period can also be recorded in other ways, for example, by a single column representing an aggregation creation time, a first measurement time, or a last measurement time.

[0077] Table schema 500 includes a column 516 that indicates which sensor (in an indicator group for a given table with schema 500) has a specific measurement in a column 518. It's important to note that in this case, a given aggregate (i.e., for a specific point in time, representing a particular write session / job of a database writer 374) can have multiple records (rows) in a table with schema 500. For example, if an indicator group for a given instance of the table schema has three indicators, A, B, and C, as shown, three rows would be used, each corresponding to an aggregate value of that indicator in the given indicator group for a given aggregate (e.g., the result of an aggregation operation, where the aggregate may contain separate aggregate values ​​for indicators within a given indicator group).

[0078] In contrast, table schema 550 is defined as containing one column for each indicator in the indicator group represented by a specific instance of the table schema. Using the example scenario provided for table schema 500, table schema 550 therefore contains columns 554, 556, and 558 for each sensor in an indicator group, with each column storing aggregate values ​​for that sensor. Using schema 550 can offer several advantages over schema 500, although both schemas provide the aforementioned benefits of adaptability (easily created / modified) and concurrency.

[0079] One problem that can occur when using the 500 table schema is that data type information can be lost if different indicators in a group provide measurements in different data types. Consider the described example with three sensors. If the sensors have the data types "float," "time," and "string" (either a string data type or a collection of fixed- or variable-length characters), storing all sensor data in a single column would typically require using a data type that can represent all these values—for example, a string / character array. In this case, however, the values ​​of individual sensors, which were not originally in string / character format, would need to be converted to a data type suitable for processing in order to use them.For example, if float values ​​for sensor B were converted to a string / array for storage in column 518, the values ​​may need to be converted back from the string / array to float values ​​for calculations or even queries (e.g., queries for values ​​greater than or less than a given value). Since column 518 can also contain a larger range of values ​​and less frequently occurring values ​​(or shorter sequences of the same value), data compression methods, such as dictionary encoding, exist to preserve column 518, thereby making the column / table less compressible.

[0080] On the other hand, by providing different columns for each sensor, the data for each sensor in an indicator group can be stored in its native data type in tables using the 550 schema, making the data more usable. Furthermore, data stored in the 550 schema can be compressed more effectively, since a given column (554, 556, 558) is likely to have a smaller range of values ​​or more frequently recurring values. This compressibility can be particularly useful when data is stored in a columnar storage format (e.g., when data for multiple records is stored by column, rather than storing all columns for a single record together).

[0081] The advantages of schema 550 do not compromise the adaptability / ease of creating indicator group tables compared to schema 500. In schema 500, when a new sensor is added to an indicator group, the new indicator and its values ​​can simply be added as entries to columns 516 and 518. When a sensor is removed from an indicator group, records for that sensor can optionally be removed from the schema 500 table. Alternatively, the data can be left in the table but logically locked for queries. For example, the indicator group definition can be used when processing a query (e.g., by linking a table of indicator group definitions to a schema 500 table). If the sensor is not included in the indicator group definition, its data will not be retrieved from the schema 500 table.

[0082] With table schema 550, new sensors can be added to an indicator group by adding a new column with the appropriate data type to the table for the sensor. When a sensor is removed, the corresponding column can be deleted from the table using table schema 550. Alternatively, as with table schema 500, when a sensor is removed from a table using schema 550, the corresponding column is not deleted from the table (at least not initially), but the data is rendered logically inaccessible, as described above.

[0083] A database writer or other component can output appropriate DDL (Data Description Language) to create and modify table schemas 500 and 550 for a given table. For example, if table schema 500 is used, a default table can be created with schema 500 because the sensor shares the same column 518 for data values, and the sensor's identity is specified in column 516. In the case of table schema 550, a CREATE TABLE DDL statement can be used to define default columns (such as timestamps associated with a specific aggregate value) and columns for each sensor in the indicator group used for table creation. When adding a sensor to a table with schema 550, an ALTER TABLE command can be used to add a column for the added sensor.If a sensor is removed from an indicator group and its data is to be deleted, an ALTER TABLE command can be issued to delete a column from the table.

[0084] In some cases, a table modified with a new column will only contain data for newly written data. For example, if a new column is added to a table with schema 550, no data values ​​will be added for existing records in the table. In other cases, NULL values ​​may be added to existing table records. In still other cases, at least when the relevant data is available, such as in a hyperscale computing system, a database writer or other component can send aggregation or data requests to the hyperscale computing system. If the relevant aggregate has already been calculated, it can be retrieved by the database writer and written to the table. If the relevant aggregate has not been calculated and the individual sensor data is available, the aggregate value can be calculated and provided to a database writer.For example, after the aggregation is complete, a message indicating that a new aggregate is available can be queued to be retrieved and processed by a database writer, as described in Example 3.

[0085] Adding sensors to an indicator group whose data is stored in a table with schema 500 can be handled analogously to schema 550. This means that in some cases, records for a given added sensor will not be added for aggregation events already included in the table. In other cases, records for an added sensor will be added for existing aggregation events, either with NULL values ​​or by determining / calculating aggregate values, as described for table schema 550.

[0086] Table schemas 500 and 550 were primarily described using columns that identify a specific aggregation event and specify how sensor values ​​are recorded for particular sensors. However, table schemas 500 and 550 can also include columns that provide other information, including values ​​corresponding to tags (also known as metadata elements, labels, attributes, properties, etc.) provided by a tagging component (e.g., tagging component 128). Fig. 1 or the tagging component 314 from Fig. 3) are added to data received from an IoT device and may also be included in aggregated values ​​for that IoT device (or a specific sensor of a specific IoT device). For example, both schemes 500 and 550 contain a column 564 for "Equipment ID," which may represent an attribute whose value is added by the tagging component 128 and 314.

[0087] Fig. Figure 5 shows a table 580 with a modified version of the table schema 550. That is, table 580 contains columns 584 and 586, which identify an aggregation period, and columns 588 (represented as 588a-588d), which represent different types of aggregate values ​​for different sensors in the indicator group represented by table 580. In particular, each sensor (11, 12, etc.) is represented with a column 588a indicating a minimum value observed during the aggregation period, a column 588b indicating a maximum value observed during the aggregation period, a column 588c indicating an average value during the aggregation period, and a column 588d indicating a number of measurements recorded during the aggregation period (not otherwise used to calculate aggregate values, such as the values ​​represented by columns 588a-588c).

[0088] Although each indicator is represented by a series of columns 588a-588d, in other embodiments different sensors may be associated with more, fewer, or different aggregate values ​​than other sensors. For example, minimum and maximum values ​​may not be relevant for a sensor that returns a Boolean value indicating a status (e.g., on or off, ready or not). Even for the same sensor type (e.g., pressure, temperature, etc.), different aggregate values ​​may be desired for different indicator groups or even individual sensors within the same indicator group. That is, minimum, maximum, and average values ​​may be desired for one temperature sensor, while for another temperature sensor only the minimum value is of interest.By adapting a table schema to the aggregate types relevant for a particular sensor, the amount of data to be stored in a given table can be reduced, as well as the effort required to generate the aggregate and the network resources needed to send the aggregate.

[0089] Table 580 contains columns 592 (represented as 592a-592c) that represent values ​​added by a tagging component when IoT messages from IoT devices are processed prior to the generation of aggregate values, for example, a column 592a containing a model identifier, a column 592b containing an equipment identifier, and a column 592c containing a template identifier, as described in Example 2.As previously explained, other tags applied to IoT data, including aggregates calculated from individual sensor values, can be used to determine which table data should be written to (including specifying that a new table is required, instantiating the table, and subsequently writing the data to the table), for example, by using a tenant identifier to identify a specific database container for the data and by identifying a specific table within that container using an indicator group identifier. Example 8 - Example of processing delayed or updated aggregates

[0090] The disclosed technologies can facilitate the processing of data from one or more sensors in an indicator group, which may be received after an aggregate for such sensors has been calculated and optionally added to a database table for the indicator group. When new sensor data arrives, an aggregate for the sensor data can be calculated, for example, using the techniques described in Example 3. Following the previously described procedure, the aggregate can be stored in a hyperscale computing system, and a notification can be queued indicating that a new aggregate is available for writing. If no previously calculated aggregate exists for the newly received sensor data, the aggregate can simply be calculated and stored as described.

[0091] If an aggregate has already been calculated for the newly received data, a new aggregate can be calculated and stored (optionally replacing the previously calculated aggregate). Recalculating an aggregate can be facilitated by retaining disaggregated data for a certain period. If the disaggregated data is unavailable, or if it is more efficient, at least some types of aggregate values ​​can be updated based on previous aggregate values. For example, minimum and maximum values ​​can be updated if newly received sensor data shows higher or lower values ​​than originally processed data. Average values ​​can be updated based on a previously recorded count, with the count being updated subsequently.

[0092] When a database writer processes an aggregate, it can first check whether at least some of the data in the aggregate represents previously written data. If so, overlaps can be resolved by updating the stored values ​​with the values ​​in the updated aggregate. If the aggregate has no overlaps with existing data, or if part of an aggregate has no overlaps with existing data, the database writer can add new records to the corresponding database table. Example 9 - Example of incorporating master data into an analytics platform

[0093] Data from IoT devices, including aggregated data, can be described as time-series data. While time-series data can provide a wealth of insights and form the basis for a variety of actions, it typically needs to be combined with other data types to enable proper processing or to support the interpretation of the results. Data that gives semantic meaning to time-series data, for example, for organizational or processing purposes, can be referred to as master data. It can be advantageous to facilitate the acquisition of relevant master data, to store the master data in a format that allows for efficient processing, and to store the data in such a way that the master data can be combined with suitable time-series data.

[0094] One advantage of the methods described in Examples 1 through 8 is that time-series data can be stored in a relational format. Master data is typically stored in a relational format and can therefore be easily processed together with the time-series data. In some cases, master data and time-series data can be stored in the same relational database system, while in other cases, data federation or other techniques can be used to retrieve data from a different database system. However, for efficient processing, it may be advantageous to store the master data and the time-series data on the same system.

[0095] It is becoming increasingly common for companies to move some or all of their data processing and storage to the cloud or to third-party environments. For example, a company might have an on-premises database system used to store master data and, optionally, other data types, such as transactional data (e.g., for an OLTP system). Particularly given the volume of data and the pressure associated with IoT applications, many companies are choosing to use third-party services for receiving, processing, and storing IoT data, including IoT data processed for storage in a relational database system.

[0096] Regardless of whether master data is stored separately from time-series data, problems can arise when combining master data with time-series data. For example, master data can be very large and stored in many tables, each with numerous attributes. Only a relatively small portion of this master data may be relevant or required for analyzing time-series data. Furthermore, master data is typically stored in a schema that is useful for other purposes, such as OLAP analysis, but inefficient for combining with time-series data and IoT data analytics.

[0097] Fig. Figure 6 shows an example of a 600 compute architecture that can be used to retrieve and store master data, including combining it with time-series data for IoT data analytics. The 600 compute architecture generally shows a variety of 610 applications (represented as 610a-610c) that communicate with a 604 cloud service (or hyperscale computing system) that provides IoT data analytics. As shown, the 610 applications and the 604 cloud service are separate systems. In other implementations, some or all of the 610 applications may reside on the 604 cloud service. Although in Fig. Since a cloud service 604 is used in 6 and the techniques described in this example 9, the disclosed techniques can be implemented analogously using a non-cloud-based system.

[0098] Each of the applications 610 comprises data 614, at least some of which is master data, a schema 616 used to store the master data, and events 618 generated by the application. At least some of the applications 610, for example, application 610a, can include objects 622. An object 622 can store or reference data 614 and optionally contain additional data that is not part of the data 614. An object 622 can be a logical data object, such as a BusinessObject, as implemented in products of SAP SE, Walldorf, Germany. Objects 622 can have specific types (e.g., customers, vendors, device types, etc.) and can be related to data 614 using techniques such as object-relational mapping. Logical data objects are described in more detail in US patent application no. 16 / 865,021, filed on May 1, 2020, which is incorporated herein by reference.

[0099] Events 618 can be used to send master data, including master data updates, to the cloud service 604. Specifically, events 618 can trigger a message 620, which is sent to, or at least made available to, the cloud service 604. A message 620 can contain data 614 relevant to an event, information that the cloud service 604 can use to obtain relevant data 614 or other data, or information about how data 614 or other data should be processed. Data 614 can influence how data 614 can be processed for a message 620 or what actions are taken in response to the message 620. For example, the message 620 can contain an identifier for the application 610 that generated the message, an event / message type, and message body that may contain relevant data 614.

[0100] A specific event type 618 can be associated with a specific message type / message content. Event types can include events 618 such as adding, deleting, or changing a schema 616 or values ​​for a specific instance of the schema (e.g., a specific table that stores data 614). Events can also be triggered due to changes to objects 622 (e.g., adding, deleting, or changing an object). An example of message 620 content might be "Add attribute X to schema Y for application Z" or "Update attribute value X for application Z to value Z".

[0101] Messages 620 can be received by a delivery service 630 of the cloud service 604. In some cases, an application 610 and the delivery service 630 can have a publisher-subscriber relationship. In other cases, the delivery service 630 can periodically query applications 610 to determine if new messages are available. In such an implementation, an application 610 can manage a message queue (not shown) that the delivery service 630 can access, for example, via an API (not shown). A suitable messaging / streaming service that can be used by the delivery service 630 and / or an application 610 is KAFKA (Apache Software Foundation).

[0102] As shown, the provisioning service 630 includes a message / streaming consumer component 634, which initially receives messages 620 from the applications 610. The messages 620 can be stored by the consumer 634 in a suitable topic (e.g., a container) 638. Topics 638 can be used to organize messages 620 based on criteria such as event / message type or message source (e.g., application 610). The provisioning service 630 also includes an interface 642 through which external components can retrieve information (e.g., messages 620 or their contents) from a specific topic 638.

[0103] Messages 620 can be retrieved from topics 638 by a collection service 644. In particular, the collection service 644 can include a consumer component 646 that regularly queries the provision service 630 for new or otherwise unprocessed messages 620. The consumer component 646 can communicate with the provision service 630 via the interface 642.

[0104] Messages 620 retrieved by the consumer component 646 can be placed in a queue 650. One or more workers 654 can remove one or more messages 620 from the queue for processing. In some cases, messages 620 are not immediately removed from the queue but are marked as pending. Messages 620 can be permanently removed from the queue when a worker 654 has finished processing a message. Workers 654 can be configured to retrieve messages 620 from the queue that are not marked as pending. In other cases, fault tolerance for the capture service 644 can be provided by logging messages 620 that are placed in or retrieved from the queue 650.The consumer component 646 can write to the log (not shown) when a message 620 is placed in the queue, the queue 650 can be configured to write to the log when a message is placed in or removed from the queue, or a worker 654 can write a log entry when it removes a message from the queue.

[0105] If a worker (654) fails to process a message (620), the message can be placed back in queue 650. Alternatively, if the message wasn't removed from the queue (but, for example, marked as being processed), its status can be changed to indicate that it is not being processed. In either case, queue 650 can track how many times a message (620) has been attempted to be processed. If a message (620) has been processed unsuccessfully a certain number of times, it can be removed from the queue, and a notification can be sent, for example, to a user or administrator, so that the error can be resolved.

[0106] One advantage of the described interaction between the applications 610, the provisioning service 630, and the collection service 644 is that it enables asynchronous, fault-tolerant processing of messages 620. This means that messages 620 can be retrieved from the provisioning service 630 as soon as workers 654 have processing capacity (which in some cases can be indicated by the number of unprocessed messages in queue 650). Similarly, workers 654 can retrieve messages 620 from queue 650 as soon as they have completed their tasks or have processing capacity.The asynchronous nature of the processing contributes to improved fault tolerance / system resilience, since, for example, if the provisioning service 630 or the collection service 644 is temporarily unavailable, the system can resume processing without data loss as soon as the respective service is available again. Fault tolerance / resilience is also increased by the fact that the queue 650 allows multiple attempts to process a specific message 620 and issues a notification if such processing repeatedly fails.

[0107] The processing of messages 620 by the workers 654 can involve various actions depending on the type / content of the message. In general, the workers 654 can perform one or more of the following actions: (1) cause data to be written to a data store 668 (e.g., a relational database system) of an analytics platform 664; (2) retrieve additional data to be written to the data store; (3) determine whether an action associated with the message is permitted; or (4) format data to be written to the data store.

[0108] As described in Example 3, data store 668 can manage both master data and time-series data (e.g., as captured according to the process described in Examples 1 to 3 or using another process). In some cases, master data can be written to data store 668 (e.g., to a table) to be used with the time-series data according to the relevant schema 616 of the application 610, which generated a message 620 associated with the data to be written. The data to be written is typically in a serialized format but can be written to a schema in data store 668 that is consistent with the corresponding schema 616. However, as described above, it can be useful to store data associated with a message 620 according to a different schema than the schema 616 from which the data may have been retrieved.In particular, it can be useful to store master data in data store 668 in such a way that efficient processing of the master data is possible, including processing the master data together with relevant time series data (e.g. using SQL JOIN operations).

[0109] In some implementations, time-series data and master data can be or conform to a star or snowflake schema, with the time-series data serving as a fact table and the master data as dimensions. Time-series data can include appropriate attributes, such as model ID, device ID, etc., to facilitate JOINs with master data, including master data that can be related to other master data (e.g., the model ID of the time-series data is joined to the model ID in a master attribute table, with one or more attributes of the master attribute table being able to be joined to tables containing additional master data).

[0110] One way to make storing master data in data store 668 more efficient is to store the data in denormalized tables. In particular, when data is stored in a columnar database, denormalization (e.g., storing master data for potentially multiple dimensions, which are stored in separate tables for an application 610, in a single table in data store 668) can reduce JOIN operations, which can be time-consuming and computationally intensive (e.g., memory and processor usage), because fewer tables need to be joined to process a query. Accordingly, when writing data to data store 668, a mapping 672 can be used to convert the master data from a source schema 616 to a target schema used with data store 668.

[0111] In some cases, a message 620 might not contain all the information a worker 654 needs to process it. For example, a worker 654 might determine that additional data, such as asset data 680 or additional master data 682, should be written along with the data 616 contained in the message 620. Alternatively, rules for processing the message 620 might require one or more actions to be performed before processing, such as verifying that a particular action is authorized. The capture service 644 can include a rule set 656. The rule set 656 can define rules for processing different types of messages 620, for processing messages based on message content (such as an application ID or information in the message body), or both.

[0112] A worker 654 can retrieve a suitable rule from the rule set 656 for a specific message 620. If the rule indicates that additional data should be retrieved and written to the data store 668, the worker can contact the appropriate data source to retrieve the required information. A data source can be an application 610, which may be the application that sent the message 620 to be processed, or another application. However, other data sources 676 can also be used, such as other databases or information storage systems, or data can be retrieved from the data store 668. The applications 610, other data sources 676, or a computing system or platform hosting such components can provide an interface, such as an API, for a worker 654 to obtain information.Alternatively, the worker 654 can communicate with the applications 610 or other data sources 676 (or the corresponding computer system or platform) via a suitable communication or data retrieval protocol, such as the SDA (Smart Data Access), SDI (Smart Data Integration), or BODS (BusinessObject Data Services) protocols used in products of SAP SE, Walldorf, Germany.

[0113] Similarly, data can be retrieved by a worker (654) from an application (610), the data store (668), or another data source (676) for use in processing a message, in addition to or instead of the data written to the data store (668). For example, authorization data (684) can be used to determine whether a particular action associated with a message, such as adding, updating, or deleting master data, is authorized for a specific application (610) or a specific user of that application that generated the event (618) that resulted in the message (620). In some cases, data retrieved by a worker (654) or written to the data store (668) to process a message (620) may be data associated with another organization or entity.Alternatively, the time-series data used to process a message 620, which is written to data store 668, may be associated with another entity. Network data 686 can be used to determine whether the data sharing resulting from processing a particular message 620 is permitted. If the authorization or network data indicates that a particular action is permitted, worker 654 can proceed. Otherwise, processing of message 620 may be aborted, optionally returning an error to the application 610 that generated the message, or to an administrator or end user associated with the time-series data linked to message 620.

[0114] Fig. Figure 6 shows a Writer 690 in the analysis platform 664, which is a database writer 374. Fig. 3. Although only a single Writer 690 is shown, the analysis platform 664 can include multiple Writers 690. The Writer 690 can write time-series data to the data store 668, as previously described. In some implementations, the Writer 690 can perform other operations, such as writing master data to the data store 668. Furthermore, the Writer 690 can optionally perform functions described as being performed by the Capture Service 664, such as converting master data from a schema 616 to a schema used in the data store 668, for example, using the mapping 672.

[0115] A query layer 694 can process queries involving time-series data, master data, or both, stored in data store 668. Query layer 694 can handle queries that perform JOIN operations between time-series data and master data, including master data that defines which indicators should have time-series data retrieved and processed. As previously described, for example, an indicator can be removed from an indicator group, rendering the data logically inaccessible for that indicator, even if the data remains stored in data store 668 for at least a certain period. Queries processed by query layer 694 can be made more efficient by using certain master data schemas, such as a denormalized schema, thus requiring fewer JOIN operations during query execution. Example 10 - Sample messages for master data changes

[0116] The Fig. 7A to 7H illustrate example messages 620 from Fig. 6, which can be sent by an application 610 and received by the provisioning service 630, in a specific implementation of the computing environment 600. Fig. Figure 7A shows a sample message 700 in JSON format for deleting a specific object, such as a particular BusinessObject (or a similar logical data object / other software object type), identified by an object ID value of 704. Note that in this case, the message body part 706, which contains data, is empty. Message 700 contains an indicator 708, which specifies a message / event type (in this case, "Delete"), identifiers 710 and 712, which identify a user and an organization, respectively, associated with an event that triggered the message, and an identifier 714 for the owner of the object to be deleted.

[0117] In some cases, a worker processing the message can use information in message 700 upon receipt, such as the identifiers 710 and 712 for the initiating user and organization, and the object ID value 704, to determine which information should be deleted from a set of master data stored for use with time-series data, for example, in data store 668 in Fig. 6. In some cases, the attributes to be deleted can be retained as part of a mapping, while in other cases, a worker may contact a software application or other data source to obtain further information about the object to be deleted (e.g., which attributes / properties it has). This information can then be used to determine which information should be deleted. The information can also be used to determine whether a particular user / organization has sufficient permissions to delete the identified object. In making this decision, a worker may request additional information, such as access permissions, from an application or other data source.

[0118] The Fig. Items 7B-7D illustrate a sample message 730 for creating an object, such as a BusinessObject (or a similar logical data object / other software object type). Message 730 contains an object ID value 734, which triggers user / organization IDs 736, 738, and an object owner ID 740. IDs 734-740 can be analogous to IDs 704, 710, 712, and 714 in message 700 and used for similar purposes (e.g., to determine whether a user / organization is authorized to create a new object / add relevant data to a data store to be used with time-series data).

[0119] Unlike content part 706 of message 700, content part 744 of message 730 contains a variety of object attribute values ​​748. When a worker processes message 730, it can determine whether all the values ​​required to add corresponding master data for use with time-series data are present in the message. If not, the worker can use a rule set to determine which additional attributes are needed and retrieve the corresponding values ​​from an application 610 or another data source. The worker can also determine which values ​​748 should be written to the data store and where they should be written, including consulting a mapping to identify an attribute in a data store table that corresponds to a value contained in content part 744.Before retrieving / writing data, a worker can determine whether the request to create an object is authorized, which can be done in a similar way to how it is described above for message 700.

[0120] The Fig. Sections 7E-7H illustrate a sample message 770 for updating an object, such as a BusinessObject (or a similar logical data object / other software object type). Message 770 can generally be similar to message 730, as it contains an object ID value 774, triggering user / organization identifiers 776 and 778, and an object owner identifier 780, which can be similar to identifiers 734-740. A main body 784 of message 770 contains a section 788, which lists old information for the object in question, and a section 792, which provides new / updated information for the object. Before applying the update, which can otherwise be similar to a "create" operation (including authentication of the action), section 788 can be used to confirm that the currently stored data (e.g.,Master data (for use with time series data) must match the data in update message 770. In some cases, an error may be triggered if the current data does not match the data in section 788 of message 770. Alternatively, the old information can be used in processes to update a data store, for example, to update a value from section 788 to a new value from section 792. Example 11 - Example schema for storing master data for IoT analysis

[0121] As described in Examples 1 and 9, a schema in which master data is originally stored may not be optimized for use with time-series data. For example, the source schema may be a star schema, which may not be well-suited for use with time-series data, such as when dealing with large volumes of time-series data. Fig. illustrate how tables 808 in a source schema 804 can be mapped to a schema 812, which is used for master data that is used with time series data, for example in data store 668 in Fig. .

[0122] As shown, tables 808 are made from Fig. 8A in Fig. 8B is denormalized to a single table 816. In other embodiments, however, the conversion of a source schema to a target schema need not involve denormalization, or even if at least some source tables 808 are denormalized, the target schema (e.g., used in data store 668) may include multiple tables implementing the source schema 804 (and optionally parts of one or more other schemas, such as other source schemas, e.g., schemas for other applications). Example 12 - Example Operations

[0123] Fig. Figure 9 is a flowchart showing a procedure 900 for interacting with a hyperscale computing system, for example, to access data stored in the hyperscale computing system. The procedure can be found in the computing architecture 300. Fig. 3 can be executed, for example through the Hyperscale interface 324.

[0124] In step 910, a call is received via an interface from an application to access data stored in a hyperscale computing system. In step 920, the call is translated into a format used by the hyperscale computing system to provide a formatted request. The formatted request is then transmitted to the hyperscale computing system in step 930. Example 13 - Computer Systems

[0125] Fig. Figure 10 shows a generalized example of a suitable computing system 1000 in which the described innovations can be implemented. The computing system 1000 is not intended to limit the scope or functionality of the present disclosure, since the innovations can be implemented in various general-purpose or specialized computing systems.

[0126] With reference to Fig. The computer system 1000 comprises one or more processing units 1010, 1015 and a memory 1020, 1025. Fig. In Figure 10, this basic configuration 1030 is shown within a dashed line. Processing units 1010 and 1015 execute computer-executable instructions, for example, to implement the technologies described in Examples 1 through 12. A processing unit can be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multiprocessor system, several processing units execute computer-executable instructions to increase processing power. For example, Figure 10 shows... Fig. 10 a central processing unit 1010 and a graphics processing unit or co-processing unit 1015. The physical memory 1020, 1025 can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or a combination of both, which the processing unit(s) 1010, 1015 can access. The memory 1020, 1025 stores software 1080 that implements one or more of the innovations described herein in the form of computer-executable instructions suitable for execution by the processing unit(s) 1010, 1015.

[0127] A computer system 1000 can have additional features. For example, the computer system 1000 includes a memory 1040, one or more input devices 1050, one or more output devices 1060, and one or more communication links 1070. A data connection mechanism (not shown), such as a bus, a control unit, or a network, connects the components of the computer system 1000. Typically, operating system software (not shown) provides an operating environment for other software running on the computer system 1000 and coordinates the activities of the computer system 1000's components.

[0128] The physical memory 1040 can be removable or non-removable and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used for the non-temporary storage of information and that can be accessed within the computing system 1000. The memory 1040 stores instructions for the software 1080, which implements one or more of the innovations described herein.

[0129] The input devices 1050 can be touch input devices such as a keyboard, mouse, stylus, or trackball, a speech input device, a scanner, or any other device that provides input to the Computer System 1000. The output devices 1060 can be a screen, printer, speaker, CD burner, or any other device that provides output from the Computer System 1000.

[0130] The 1070 communication link(s) enable communication with another computing unit via a communication medium. The communication medium transmits information such as computer-executable commands, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal in which one or more properties are set or modified to encode information within the signal. For example, communication media can use electrical, optical, RF, or other carriers, but this is not a limitation.

[0131] These innovations can be described in the general context of computer-executable instructions, such as those contained in program modules that run on a real or virtual target processor within a computing system. Generally, program modules or components comprise routines, programs, libraries, objects, classes, components, data structures, and so on, which perform specific tasks or implement certain abstract data types. The functionality of program modules can be combined or distributed across various modules as desired. Computer-executable instructions for program modules can be run within a local or distributed computing system.

[0132] The terms "system" and "device" are used synonymously here. Unless the context clearly indicates otherwise, neither term implies a restriction to a particular type of computing system or device. In general, a computing system or device can be local or distributed and can include any combination of specialized and / or general-purpose hardware with software that implements the functionality described herein.

[0133] In various examples described herein, a module (e.g., a component or an engine) can be "coded" to perform certain operations or provide certain functions, meaning that computer-executable instructions can be run on the module to perform such operations, have such operations performed, or otherwise provide such functions. Although the functionality described in respect of a software component, module, or engine can be implemented as a discrete software unit (e.g., program, function, class method), it need not be implemented as a discrete unit. That is, the functionality can be integrated into a larger or more general program, for example, into one or more lines of code within a larger or more general program.

[0134] For the sake of clarity, the detailed description uses terms such as "determine" and "use" to describe computer operations within a computing system. These terms are highly abstract concepts for operations performed by a computer and should not be confused with human actions. The actual computer operations corresponding to these terms vary depending on the implementation. Example 14 - Cloud Computing Environment

[0135] Fig. Figure 11 shows an example of a cloud computing environment 1100 in which the described technologies can be implemented. The cloud computing environment 1100 comprises cloud computing services 1110. The cloud computing services 1110 can include various types of cloud computing resources, such as computer servers, data storage, network resources, etc. The cloud computing services 1110 can be centralized (e.g., provided by a company's or organization's data center) or distributed (e.g., by different computing resources at different locations, such as different data centers and / or in different cities or countries).

[0136] Cloud computing services 1110 are used by various types of computing devices (e.g., client computing devices) such as computing devices 1120, 1122, and 1124. For example, computing devices (e.g., 1120, 1122, and 1124) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smartphones), or other types of computing devices. For example, computing devices (e.g., 1120, 1122, and 1124) can use cloud computing services 1110 to perform computing operations (e.g., data processing, data storage, and the like). Example 15 - Implementations

[0137] Although the operations of some of the disclosed methods are described in a specific order for clarity, it should be clear that this type of description also includes rearrangement, unless a specific order is prescribed by the language specified below. For example, operations described sequentially may, in some cases, be rearranged or performed concurrently. Furthermore, for the sake of simplicity, the accompanying figures may not show the various ways in which the disclosed methods can be used in conjunction with other methods.

[0138] Each of the disclosed methods can be implemented as computer-executable instructions or as a computer program product stored on one or more computer-readable storage media, such as physical, non-volatile computer-readable storage media, and executed on a computing device (e.g., any available computing device, including smartphones or other mobile devices containing computer hardware). Physical computer-readable storage media are any available physical media accessible within a computing environment (e.g., one or more optical media discs, such as DVDs or CDs, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as flash memory or hard disks)). For example, and with reference to Fig.10. Computer-readable storage media include the 1020 and 1025 memory types, as well as the 1040 memory type. The term "computer-readable storage media" does not include signals and carrier waves. Furthermore, the term "computer-readable storage media" does not include communication links (e.g., 1070).

[0139] All computer-executable instructions for implementing the disclosed techniques, as well as all data created and used during the implementation of the disclosed embodiments, can be stored on one or more computer-readable storage media. The computer-executable instructions can, for example, be part of a dedicated software application or a software application accessed or downloaded via a web browser or other software application (e.g., a remote computing application). Such software can be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or another such network) using one or more network computers.

[0140] For the sake of clarity, only certain selected aspects of the software-based implementations are described. Other details known in the field are omitted. It is understood, for example, that the disclosed technology is not limited to any particular programming language or program. For instance, the disclosed technology may be implemented by software written in C, C++, C#, Java, Perl, JavaScript, Python, Ruby, ABAP, SQL, Xcode, Go, Adobe Flash, or any other suitable programming language, or, in some examples, by markup languages ​​such as HTML or XML, or combinations of suitable programming and markup languages. Likewise, the disclosed technology is not limited to any particular computer or type of hardware. Certain details of suitable computers and hardware are generally known and need not be detailed in this disclosure.

[0141] Furthermore, each of the software-based embodiments (which, for example, include computer-executable instructions that cause a computer to execute one of the disclosed methods) can be uploaded, downloaded, or remotely accessed via a suitable means of communication. Such suitable means of communication include, for example, the Internet, the World Wide Web, an intranet, software applications, cables (including fiber optic cables), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such means of communication.

[0142] The disclosed methods, devices, and systems should not be interpreted as restrictive in any way. Instead, the present disclosure relates to all novel and non-obvious features and aspects of the various disclosed embodiments, alone and in various combinations and subcombinations with one another. The disclosed methods, devices, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments require that one or more particular advantages be present or problems be solved.

[0143] The technologies from one example can be combined with the technologies described in one or more of the other examples. Given the many possible embodiments to which the principles of the disclosed technology can be applied, it should be noted that the embodiments shown are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology. Rather, the scope of the disclosed technology includes what is covered by the scope and spirit of the following claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 16 / 865,021

[0098]

Claims

[1] A system for storing aggregated sensor data in a relational database, comprising: a storage facility; one or more processing units coupled to the memory; and one or more computer-readable storage media that store instructions which, when loaded into memory, cause one or more processing units to perform operations that include: by means of a data collection service (120, 312) periodically reading sensor data from a plurality of devices from a streaming service (114), wherein each device of the plurality of devices comprises one or more hardware sensors; Determining a set of attributes or metadata elements to be applied to the sensor data and annotating the sensor data by a tagging component to produce tagged data, wherein the annotation of the sensor data includes adding metadata to the sensor data or editing metadata of the sensor data, wherein the one or more hardware sensors are grouped into one or more indicator groups, and wherein the tagged data is stored as time series data in a data store; by an aggregation engine (356), aggregating data for a group by generating one or more aggregated values ​​from the tagged data stored in the data store; through an aggregation engine (356), writing the aggregated values ​​as aggregates in a semi-structured format back into data storage; and by a database writer (374), instantiating a relational database table, wherein the relational database table comprises for each respective hardware sensor in an indicator group a respective column in which aggregate values ​​for the respective hardware sensor are stored; and Calling a data read method to retrieve data for the aggregates and writing the aggregates to the relational database table. [2] System according to claim 1, wherein generating the one or more aggregated values ​​comprises generating aggregated values ​​over a period of time represented by the time series data being processed. [3] The system according to claim 2, wherein the one or more aggregated values ​​generated by the aggregation engine comprise minimum or maximum values ​​over the period. [4] The system according to claim 2, wherein the one or more aggregated values ​​generated by the aggregation engine comprise a count value for the period. [5] The system according to claim 1, wherein a new sensor is added to an indicator group and a new indicator and its values ​​are added as entries to columns of the relational database table. [6] The system according to claim 1, wherein the relational database table contains a column that provides time information for a specific row of the table. [7] The system according to claim 1, wherein the multiple devices comprise multiple Internet of Things (IoT) devices. [8] A data collection service system comprising a storage unit and one or more processing units, wherein the data collection service system is configured to perform a procedure comprising: Receiving Internet of Things (IoT) data from a multitude of IoT devices, where each IoT device in the multitude of IoT devices comprises one or more hardware sensors; Defining a set of attributes or metadata elements to be applied to the IoT data and annotating the IoT data by a tagging component to generate tagged data, wherein annotating the IoT data includes adding metadata to the IoT data or editing metadata of the IoT data, wherein the one or more hardware sensors are grouped into one or more groups, and wherein the tagged data is stored as time-series data in a data store in a semi-structured format; Aggregating data for a specific group by generating one or more aggregated values ​​from the tagged data stored in the data store; Writing the aggregated values ​​back to the data storage in the semi-structured format; Instantiating a relational database table, wherein the relational database table includes, for each respective hardware sensor in a given group, a respective column configured to store aggregated values ​​associated with that hardware sensor; and Calling a data read method to retrieve the aggregated values ​​and write the aggregated values ​​to the relational database table. [9] The data acquisition system according to claim 8, wherein generating the one or more aggregated values ​​comprises generating aggregated values ​​over a period of time represented by the time series data to be processed. [10] The data collection service system according to claim 9, wherein the one or more aggregated values ​​generated by the aggregation engine comprise minimum or maximum values ​​over the period. [11] The data collection service system according to claim 9, wherein the one or more aggregated values ​​generated by the aggregation engine comprise a count value for the period. [12] The data collection service system according to claim 8, wherein a new sensor is added to an indicator group and a new indicator and its values ​​are added as entries to columns of the relational database table. [13] The data collection service system according to claim 8, wherein the relational database table contains a column that provides time information for a specific row of the table. [14] The data acquisition service system according to claim 13, wherein the time information includes start and stop timestamps, an aggregate generation time, a first read time or a last read time. [15] The data collection service system according to claim 8, wherein the IoT data is sent to a streaming service and wherein data in the streaming service is regularly read by the data collection service system.

Images