Managed data export from edge devices to remote networks
By implementing a managed data export manager on edge devices, the problem of network connectivity changes in IoT device data transmission is solved, and the efficiency and reliability of data transmission is achieved, ensuring application performance and data security.
Patent Information
- Application Number
- CN202080081092.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-12-12
- Filing Date
- 2020-11-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-11-20
AI Technical Summary
In the process of exporting large amounts of data generated in IoT devices, it is necessary to deal with changing network connectivity. It is difficult for the existing technology to effectively manage and prioritize data transmission, resulting in data transmission efficiency and reliability issues.
Automatically handle network connectivity changes by implementing a managed data export manager on edge devices, qualifying and sending data streams with data flow configuration and priority scheduling techniques, ensuring priority transmission of critical data and pausing and recovery when the network is unavailable Data export.
Improve the reliability and efficiency of data transmission, dynamically adjust data transmission strategies to adapt to network changes, and ensure application performance and data security.
Smart Images

Figure CN114731342B_ABST
Abstract
Description
Background Art
[0001] With the increased availability of different smart devices that access the Internet, many new applications have been developed that take advantage of Internet connectivity. The Internet of Things (IoT) is a phrase given to the interconnection of computing devices dispersed around the globe within the existing Internet infrastructure. IoT devices can be embedded in a variety of products such as home appliances, manufacturing equipment, printers, cars, thermostats, smart traffic lights, environmental sensors, etc.
[0002] Since a large amount of data is generated in numerous IoT devices, it may be helpful to aggregate the generated data into cloud services. For example, data from many different IoT devices can be processed by cloud services using data processing techniques to perform analysis or perform various functions in response to the received data. However, exporting data from different locations to remote services may have to cope with changing environmental conditions, such as changing network connectivity to send data to remote services. Therefore, technologies that improve the ability to export data to remove services from different devices are highly desirable. BRIEF DESCRIPTION OF THE DRAWINGS
[0003] Figure 1 A series of logical block diagrams are shown for managed data export from an edge device to a remote network in accordance with some embodiments.
[0004] Figure 2 An example provider network is shown that can implement a service that can receive data exported by a local execution environment in a client network in accordance with some embodiments.
[0005] Figure 3 A logical block diagram of a data export manager according to some embodiments is shown.
[0006] Figure 4 A logical block diagram is shown illustrating the interactions for configuring a managed export for a data flow in a local execution environment according to some embodiments.
[0007] Figure 5 A high-level flow chart illustrating various methods and techniques for implementing managed data export from an edge device to a remote network, according to some embodiments.
[0008] Figure 6 A high-level flow chart is shown for implementing various methods and techniques for evaluating network connectivity criteria to derive data from a data stream at an edge device in accordance with some embodiments.
[0009] Figure 7 A high-level flow diagram is shown for implementing various methods and techniques for storing data received from a data flow at an edge device for export to a target service or another data flow destination, in accordance with some embodiments.
[0010] Figure 8 An example system for implementing the various methods, techniques, and systems described herein is shown in accordance with some embodiments.
[0011] Although embodiments are described herein for several embodiments and illustrative drawings by way of example, it will be appreciated by those skilled in the art that embodiments are not limited to the described embodiments or drawings. It should be understood that the drawings and detailed descriptions thereof are not intended to limit the embodiments to the disclosed specific forms, but on the contrary, it is intended to cover all modifications, equivalents and alternatives falling within the spirit and scope defined by the appended claims. The titles used herein are only used for organizational purposes and are not meant to limit the scope of the specification or claims. As used throughout this application, the word "may" is used in an allowed sense (i.e., meaning "possible") rather than a mandatory sense (i.e., meaning "must"). Similarly, the words "include", "including" and "includes" are meant to include but are not limited to.
[0012] It should also be understood that, although the words first, second, etc. can be used to describe various elements in this article, these elements should not be limited by these words. These words are only used to distinguish one element from another element. For example, without departing from the scope of the present invention, the first contact can be referred to as the second contact, and similarly, the second contact can be referred to as the first contact. Both the first contact and the second contact are contacts, but they are not the same contact. DETAILED DESCRIPTION
[0013] Various techniques for managed data export from edge devices to remote networks are described herein. The remote network provides various storage, computing, and other services that various client networks, systems, applications, or devices can utilize to complete, enhance, control, or otherwise interact with different services or applications implemented on the client. In addition, the remote network provides scalable resources, thereby providing higher persistent availability for applications on the client. Thus, in various scenarios, data generated on the client, such as various types of machine-generated data (e.g., sensor readings) or manually generated data (e.g., user interface operations such as voice, touch, or other input) can be transmitted to one (or more) services of the remote network as part of executing an application.
[0014] Edge devices, such as Figure 8The computing device 1000 discussed herein may, in some embodiments, be implemented to serve as an access and / or coordination point for interactions between a client and a remote network in various embodiments. For example, an edge device may be a remote network (such as, for example, Figure 2 The client may send data to an edge device in order to utilize features and / or functionality of the edge device, such as the local execution environment of the service of the remote network, to implement various features of the application that would otherwise only be executed at the service of the remote network. The local execution environment may then selectively determine whether further operations or requests should be performed at the service of the remote network.
[0015] In scenarios where data from clients is to be stored, processed, or otherwise handled in a service in a remote network, the edge device can implement various techniques to manage the export of data to a target service or another data flow destination in the remote network. Given the unpredictable nature of network connectivity, the edge device can implement managed data export to handle data export on behalf of different clients that results in varying network connectivity. For example, in scenarios where bandwidth capacity to send data is limited, managed data export can automatically prioritize which data to export when some data may be more important than other data (e.g., an alarm event may be more important than a single sensor reading).
[0016] Figure 1 A series of logical block diagrams for managed data export from an edge device to a remote network are shown in accordance with some embodiments. The remote network 120 (which may be similar to the provider network 200 or other types of remote networks) may implement one or more services 140 to accept data generated at different devices away from the remote network for storage, analysis, or other processing. For example, different devices, such as devices 142a, 142b, and 142c (e.g., various computing or electronic devices that can receive, capture, or generate data and transmit data, such as one or more computing devices 1000 or other devices with embedded processing and network capabilities) may send data as a data stream (e.g., including various objects, chunks, messages, records, or other portions of data) to the service 140.
[0017] Client network 110 (e.g., an on-premise network, a private network, a virtual private network, or other network physically or logically isolated from remote network 120) can facilitate communication to send data stream 152 to another device, such as edge device 130 (e.g., hereinafter Figure 8Rather than imposing the cost of implementing direct communication for each of these devices 142, edge devices 130 may implement various connections, protocols, interfaces, and other features for communicating with services 140. In various embodiments, edge devices 130 may implement data export management 132 to handle receiving and sending data streams from devices 142 to services 140 in remote networks 120.
[0018] As described above, the network connection between the edge device 130 and the service 140 in the remote network 120 may change. In various embodiments, the data export management 132 can automatically handle these and other issues associated with exporting data to the service 140. For example, the data export management 132 can handle issues related to network instability by pausing and preserving the data export state of each stream when the connection is unavailable (e.g., through checkpoints) and resuming the export when the connection becomes available again. Because different data streams may be using and / or competing for network connection bandwidth, the data export management 132 can arbitrate and send data from different data streams, as shown at 154 (e.g., where "A" data may come from a data stream from device 142a, "B" data may come from a data stream from device 142b, and "C" data may come from a data stream from device 142c). In various embodiments, a developer, administrator, or other responsible party can configure the data export management 132 by defining, enabling, uploading, and / or otherwise providing export configurations for data streams, as described below with respect to Figure 4 The export configuration of a data flow may inform arbitration between data flows and / or the environment in which the data flow is sent, as discussed below with respect to Figure 3 and Figures 5 to 7 Discussed in detail.
[0019] For example, as shown in scenario 102, corresponding portions "A", "B", and "C" of different data streams are equally interleaved for transmission to service 140. Data export management 132 can implement a first-in-first-out (FIFO) scheduling technique (e.g., by implementing a FIFO queue) that can send a portion of the received data stream. It should be noted that in some embodiments, each data stream can be bound to the same or different corresponding services.
[0020] As shown in scenario 104, a network connection for sending a data stream from device 142 in client network 110 may be unavailable, as shown at 162. However, device 142 may still be receiving and / or generating additional data to send as corresponding data stream 152. Data export management 132 may implement techniques to handle the lack of network connectivity, including various storage strategies, including according to the following description of Figure 7 The discussed techniques are used to preserve data stream storage. Portions of the data may be dropped or otherwise not stored, for example if storage capacity needs to be reserved for more important data.
[0021] As shown in scenario 106, a network connection may become available. Data export management 132 may send different data streams of data, as shown in 172. In some scenarios, a priority or other selection scheme may be applied to determine the order in which a portion of the data should be sent. For example, in the example shown, portion "B" of the data (which may have been stored for transmission with a portion of the "A" and "C" data streams) is selected before portions "A" and "C" are sent. As described below with respect to Figure 3 and Figure 5 As discussed in detail, various selection techniques may be applied that may be configured or implemented using export configurations for data streams. For example, different priority values may be assigned to different data streams in order to implement priority scheduling. Supporting configurable export data streams allows the performance of data export between different client devices 142 to adapt to the design and performance requirements of applications running in the client network 110 while retaining the benefits of automated data export management. The data export management 132 may, for example, dynamically adjust which data streams are sent in various situations, including changes in network connectivity, thereby improving the performance of applications implemented on devices 142 utilizing the service 140. In at least some embodiments, the export configuration may include encoding or decoding features that may be specified (e.g., via a script or other programming language, library, or executable application) to customize the export of individual data streams bound to different types of services.
[0022] It should be noted that the foregoing description of managed data export from an edge device to a remote network is a logical description and, therefore, should not be construed as limited to implementing remote networks, services, network-connected client networks, client devices, edge devices, or data flows (or portions thereof), including the number or combination of hardware and / or software used to implement such features.
[0023] This specification first provides a general description of a provider network that implements a variety of different services, including Internet of Things (IoT) services and / or other services that may be the target of or involve the deployment or implementation of managed data export from edge devices and client networks to provider networks, with the provider network having a local execution environment that can implement data export management. Various examples that can be used as part of implementing data export management are then discussed, including different components / modules, or arrangements of components / modules. Many different methods and techniques for implementing managed data export from edge devices to remote networks are then discussed, some of which are shown in the accompanying flowcharts. Finally, a description of an example computing system on which various components, modules, systems, devices, and / or nodes can be implemented is provided. Various examples are provided throughout the specification.
[0024] Figure 2 An example provider network is shown that can implement services that can receive data exported by local execution environments in client networks in accordance with some embodiments. In one embodiment, provider network 200 can be a dedicated or closed system, or can be established by an entity such as a company or public sector organization to provide one or more services (such as various types of cloud-based storage) to clients 250 that can be accessed via the Internet and / or other networks. In one embodiment, provider network 200 can be implemented in a single location, or can include numerous data centers that host various resource pools required to implement and distribute the infrastructure and services provided by provider network 200, such as a collection of physical and / or virtualized computer servers, storage devices, networking devices, etc. (e.g., see below with respect to Figure 8 1000). In some embodiments, provider network 200 may implement various computing resources or services, such as IoT services 210, data storage services 220, data streaming services 230, and / or any other type of network-based services 240 (which, in some embodiments, may include virtual computing services and various other types of storage, processing, analysis, communication, event processing, visualization, data cataloging, data acquisition (e.g., ETL), and security services).
[0025] In various embodiments, Figure 2 The components shown may be implemented directly in computer hardware as instructions that are directly or indirectly executable by computer hardware (e.g., a microprocessor or computer system), or using a combination of these techniques. Figure 2 The components of can be implemented by a system including a certain number of computing nodes (or simply referred to as nodes), each of which can be similar to Figure 8The computer system embodiments shown and described below. In various embodiments, the functionality of a given system or service component (e.g., a component of IoT service 210) can be implemented by a specific node, or can be distributed across several nodes. In some embodiments, a given node can implement the functionality of more than one service system component (e.g., more than one data storage component).
[0026] The IoT service 210 can deploy, coordinate, and provide various IoT-based applications that utilize various remote devices (e.g., client devices 252 and 254 in the client network 250). For example, different client devices implemented in different locations, networks, and other scenarios can collect, generate, and interoperate to execute different applications. The IoT service 210 can coordinate the interaction of client devices with other services, such as data storage services 220, data streaming services 230, or other services 240 (e.g., artificial intelligence or other services that can be analyzed as a whole to provide control or other information to client devices). The IoT service 210 can provide enforceable security policies or controls to ensure data privacy, prevent malicious behavior, and meet various industry or government regulatory systems. Because the IoT service 210 is implemented as part of the provider network, the IoT service 210 can provide the scalability, availability, persistence, and other benefits of the provider network (e.g., cloud service benefits) to enable client applications to have appropriately sized resources to meet demand.
[0027] One feature of the IoT service 210 that client devices can utilize is a control service 212. In various embodiments, the control service 212 can coordinate the deployment of IoT applications across devices and networks to provide secure interactions between different client devices. The control service 212 can be integrated or connected with artificial intelligence and other data analysis technologies or services for training and applying machine learning models to add or improve application features of client devices. In some embodiments, the control service 210 can monitor and / or audit client device applications to enforce or review security policies or other controls. The control service 212 can provide remote management, IoT application updates, changes, or other tasks or operations for client devices to be performed without physical access to the client device.
[0028] In some embodiments, IoT service 210 may also implement data analysis 213. Data analysis 213 may include various analytical tools, such as tools for processing structured and unstructured data. Data analysis 213 may, for example, provide features for filtering and converting data streams from client devices into other formats that are easier to analyze, such as for creating time series data to be stored in a time series database, machine learning data for updating or training machine learning models, or various other analyses (e.g., supporting queries or other requests to obtain data from a data stream (or group of data streams). In some embodiments, data analysis 213 may be a fully managed feature that scales up or down to handle different loads for receiving, filtering, converting and / or storing data streams from client devices. In some embodiments, data analysis 213 may monitor data streams to detect events and respond to events.
[0029] In various embodiments, IoT service 210 may implement deployment 214. For example, deployment 214 may provide access to configure and deploy software applications and / or operating systems that support and / or implement IoT applications at the edge of a provider network in a client network, such as client network 250. For example, deployment 214 may provide an operating system for a microcontroller or other low-power edge device that implements network connectivity and other features that support, for example, interaction with other devices in client network 250 without occupying limited processing resources that are typically available in such devices. In various embodiments, deployment 210 may provide a local execution environment, such as local execution environment 255 (or Figure 1 As described below with respect to the edge device 130 in Figure 4 As discussed, data export management, such as data export management 256 (or data export management 132 ), may be configured and deployed by deployment 214 .
[0030] Edge devices, such as Figure 1 and Figure 2 As shown in and below regarding Figures 5 to 7Edge devices such as those discussed, in various embodiments, can provide execution environments to perform various IoT services 210 and other provider network 200 service features that can be performed without network connectivity. For example, a local execution environment hosted on an edge device can support programming or interaction with various event-driven functions or operations, messaging platforms, security functions, and provider network 200 services, third-party applications, or local (e.g., pre-built) applications connected to a client network. In this way, in some embodiments, an application implemented on a local device can connect to and utilize the local execution environment to perform various operations, receive instructions, report data, or other application tasks, just as if the local execution environment were the provider network 200. As described below with respect to Figure 3 As discussed in detail, managed data export, such as data export management 256 (or Figure 1 Data export management 132) in can be implemented as part of the local execution environment on the edge device.
[0031] In various embodiments, the IoT service 210 may implement an interface 211. The interface 211 may be one or more types of interfaces for accessing IoT services to programmatically and perform control or management operations on behalf of a user. For example, the interface 211 may include one or more programming interfaces (e.g., application programming interfaces (APIs)) that allow client devices to call or request various IoT service operations (e.g., call data analysis 213 or report monitoring information for control service 212). In some embodiments, the interface 211 may include a command or console, implemented as a command line interface and / or a graphical user interface (e.g., implemented as a web-based console) to configure or execute various IoT service features, such as those described below with respect to Figure 4 The data export configuration characteristics discussed.
[0032] In various embodiments, the provider network 200 may implement other services that may be the target or destination of the data being exported by the data export management 256. In some embodiments, the data storage service 220 may implement different types of data storage components for storing, accessing, and managing data on behalf of the client as network-based services, enabling the client to operate a data storage system in a cloud or network computing environment. For example, the data storage service 230 may include various types of database storage services (relational and non-relational) for storing, querying, and updating data. Such services may be highly scalable and extensible enterprise-level database systems. Queries may be directed to databases in the data storage service 220 distributed across multiple physical resources, and the database system may be scaled up or down as needed. In various embodiments, the database system may effectively collaborate with database schemas of various types and / or organizations. In some embodiments, the client / subscriber may submit queries in a variety of ways, for example, interactively submitting to the database system via a SQL interface. In other embodiments, external applications and programs may submit queries to the database system using an open database connectivity (ODBC) and / or Java database connectivity (JDBC) driver interface.
[0033] In some embodiments, a data storage service 220 may be implemented as a centralized data storage so that other data storage services can access data stored in the centralized data storage for processing and / or storage in other data storage services. A may provide storage and access to various object or file data storages to add, update, and obtain data objects or files of various types and sizes or collections of data objects or files. Such data storage services 220 may be accessed through a programming interface (e.g., an API) or a graphical user interface. The centralized data storage may provide virtual block-based storage for maintaining data as part of a data volume, which may be mounted or accessed similarly to a local block-based storage device (e.g., a hard drive, a solid-state drive, etc.), and may be accessed using a block-based data storage protocol or interface (such as an Internet Small Computer Interface (iSCSI)).
[0034] In at least some embodiments, one of the data storage services 220 may be a data warehousing service that utilizes a centralized data storage implemented as part of another data storage service 220. The data warehousing service may provide a variety of different data management services to the client according to the various needs of the client. In some cases, the client may wish to store and maintain a large amount of data, such as sales record marketing, management reports, business process management, budget forecasts, financial reports, website analysis, or many other types or categories of data. The client's use of data may also affect the configuration of the data management system used to store data. For example, for certain types of data analysis and other operations (such as those operations that aggregate large data sets from a small number of columns within each row), a columnar database table can provide more efficient performance. In other words, the column information from the database table can be stored in data blocks on the disk, rather than storing the entire row of the column in each data block (as in traditional database solutions).
[0035] Another service in the provider network 200 may be a data stream service 230. The data stream service 230 may provide a programming interface (e.g., an application programming interface (API), a web page or website, a graphical user interface, or a command line tool) to enable the creation, configuration, and deletion of streams. In some embodiments, the programming interface may also enable the submission, storage, analysis, conversion, and / or retrieval of stream data records. In various embodiments, some clients of the stream management system may develop applications that directly call the stream management programming interface. However, in at least some embodiments, in addition to the stream management interface, a higher level abstraction or application level processing framework may be provided to customers, which may simplify various aspects of stream processing for customers who do not wish to develop applications using the lower level stream management functionality supported by the stream management system. Such a framework may provide its own programming interface (e.g., built on top of the stream management system interface), allowing customers to focus more on the business logic implemented using the stream records rather than the lower level stream management operations. The higher level framework may be implemented as part of the stream management system to provide fully managed data streams.
[0036] For example, the data stream service 230 may implement a data acquisition system that is configured to obtain data records for a particular data stream from a data producer (e.g., by operating one or more acquisition nodes for the data stream). In some embodiments, the data records for a stream may be obtained according to a scheme for partitioning the data stream. The partitioning scheme may be selected by a client of the data stream service 230 for a data stream so as to receive data records from a data producer indicating a particular partition to which the data record belongs. However, in some embodiments, the data stream may be fully managed by the data stream service 230, and the data producer may send data records without any partitioning direction. Instead, the data acquisition system may allocate data records to route the data records to identified partitions. Once acquired, the stream management service may store the acquired data records (e.g., on a corresponding storage node provided for the data stream). Such storage nodes may record, save, store, or otherwise persist the data records on any of a variety of types of storage devices (which may be performed according to the persistence policy of the data stream).
[0037] To retrieve data from a data stream, the stream management service may provide a retrieval system (e.g., implement a retrieval node) that can access stored data records of the data stream. In some embodiments, data retrieval may be performed in response to a request from a consumer (e.g., a stream processing node that performs processing on the data stream data).
[0038] In some embodiments, the data stream service 230 may provide an interface that supports one or more sets of programming interfaces (e.g., an application programming interface (API), a web page or website, a graphical user interface, or a command line tool) to enable the creation, configuration, and deletion of data streams (client-hosted or fully-hosted), as well as the submission, storage, and retrieval of stream data records. For example, a data producer may be configured to add a data record to a data stream by utilizing a client library provided by the data stream service 230 to utilize a request to send a "putRecord" request to the data stream service 230 through an interface. Similarly, a data consumer may be configured to access the data stream service 230 through an interface and utilize a client library provided by the data stream service 230 to "getNextRecords" when executing an application to retrieve the next data record to be processed in the data stream.
[0039] In some embodiments, other services 240 may also include services that are the target of managed exports of data. For example, other services 240 may include data processing services that may be of various types to perform different functions (e.g., query or other processing engines to perform functions such as anomaly detection, machine learning, data lookup, or any other type of data processing operation). For example, in at least some embodiments, data processing services may include a map reduce service that creates a cluster of processing nodes that perform map reduce functions on data stored in one of the data storage services 220. Various other distributed processing architectures and techniques may be implemented through data processing services (e.g., grid computing, sharding, distributed hashing, etc.).
[0040] In general, clients of the provider network can access the provider network 200 through the network 260. In some embodiments, such clients can transmit network-based service requests (e.g., requests to send data from a data stream through the network 260). In various embodiments, the network 260 can cover any suitable combination of networking hardware and protocols necessary for establishing network-based communications between the client and the provider network 200. For example, the network 260 can generally cover various telecommunication networks and service providers that jointly implement the Internet. In one embodiment, the network 260 can also include a private network, such as a local area network (LAN) or a wide area network (WAN) and a public or private wireless network. In this embodiment, the network 260 can include hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewall / security software, etc.) necessary for establishing a networking link between a given client and the Internet and between the Internet and the provider network 200.
[0041] In various embodiments, client network 250 may transmit communications between client device 252 and client device 254. Similar to network 260, client network 250 may include the necessary hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewall / security software, etc.) to establish a networking link between client device 252 and client device 254.
[0042] Client device 252 may be a computing device, such as the one described below with respect to Figure 8 The computing system 1000 discussed. In some embodiments, the client device 252 can be a variety of sensors, appliances, instruments, or other devices that can implement a microcontroller or can capture, generate, and send data (e.g., as described above with respect to Figure 1The data flows discussed) to other client devices 252 and other processing components of client devices 254 in client network 250. Client devices 254 can be various types of computing devices, such as, Figure 8 The computing system 1000 in the embodiment of the present invention, or other edge devices, such as various types of gateway devices.
[0043] The client device 254 may implement a local execution environment 255 and a data export management 256 to provide managed export of data streams. Figure 3 256 can be configured by and / or use a data stream export configuration 360 specified by a client device (as described below with respect to Figure 4 discussed), which can specify the processing of data streams received and processed by data export management 256.
[0044] For example, the data flow export configuration 360 may specify a priority value, label, or other designation that the data flow admission 310 may use when receiving a data flow record, as shown at 302. For example, when the storage capacity of the data flow record reaches a threshold utilization (e.g., when the data flow export queue 320 exceeds the number of entries or the storage size is greater than a size threshold), the data flow admission 310 may implement throttling or other resource conservation techniques based on the designated priority of the data flow, as described below with respect to Figure 7 Such techniques may be enabled, disabled (or different thresholds may be applied) based on network connection status information, which may be determined by network connection monitoring 350 and provided to data stream admission 310. For example, if the network connection is active, data stream admission may not restrict or prioritize admission of certain data streams. If the network connection is degraded or not running at full bandwidth capacity, some throttling based on high utilization thresholds may be applied, and if the network connection is disconnected or otherwise unavailable, a lower utilization threshold before throttling may be applied (e.g., to prevent overwriting of data when storage capacity is exceeded).
[0045] The data stream admission 310 may also apply an export configuration 360 to the format or information stored in the data stream export queue 320. For example, the data stream export configuration may specify unnecessary or necessary data to be removed, or additional data features to be added to the data in the stream before being stored in the data stream export queue (e.g., adding a prefix, timestamp, identifier, or other metadata). In some embodiments, the data stream export configuration 360 may specify the allocation or size of the data stream export queue 320 (or other storage structure) for the data stream, and once filled, the data of the data stream is blocked or rejected.
[0046] In some embodiments, data export management 256 may implement a data stream export queue 320. Data stream export queue 320 may be implemented to provide an order for selecting data records from different streams (e.g., a FIFO or priority queue), or may be a buffer or other allocated storage space that stores records selected by data stream export selection 330.
[0047] In various embodiments, data stream export selection 330 can be implemented by data export management 356 to provide a fully managed data export feature for received data streams. Data stream export selection 330 can utilize the data stream configuration to perform various analyses to select data streams for export. For example, priority analysis 332 can be performed to select different specified data stream priorities (e.g., "high", "medium", or "low"). In some embodiments, priority analysis 332 can be implemented to identify and send higher priority data, which will also increase or compensate for the priority of waiting data stream records so that no data stream will starve other data streams. Data stream export selection 330 can implement network analysis 334, as described below with respect to Figure 6 As discussed in detail, various types of selections may be performed based on the state of the export network connection 304, which may be provided by network connection monitoring. Other selection analyses 336 or techniques, such as last in first out (LIFO), round robin, or other selection techniques, may be performed as specified or informed by the data flow export configuration 260.
[0048] In some embodiments, data stream export selection may operate differently depending on the state of the export network connection 304. For example, if the export network connection is healthy or otherwise operating at full capacity, then FIFO ordering or balanced selection techniques may be implemented. However, if the network connection has just been reestablished (e.g., after a failure), then as described above with respect to Figure 1 and the following about Figure 5 As discussed, selection techniques may selectively choose for which data streams to send data (eg, based on priority).
[0049] like Figure 3As shown, in some embodiments, the data export management can implement different destination connectors, such as destination connectors 340a, 340b, and 340c. The destination connector 340 can format and send the data from the stream to the target service or another data stream destination formatted according to the interface at the destination, as shown in 306. For example, the destination connector 340a can format a request for storing the data stream data as different entries in a database table, while the destination connector 340b can form a request to add the data stream record as an entry in the data stream service, that is, the data stream service 230. In some embodiments, the destination connector can be provided, configured, or uploaded by the user to be included in the data export management 256 when the local execution environment is deployed.
[0050] As previously described, in some embodiments, the IoT service 210 may implement a deployment feature, deploy 214 , to deploy the local execution environment and the data export manager to the edge device. Figure 4 A logical block diagram is shown illustrating the interactions for configuring a managed export for a data flow in a local execution environment according to some embodiments. Figure 4 As shown, IoT deployment 214 can maintain local execution environment configuration 402. Local execution environment configuration 402 can include data export management (as discussed above) and other features. In various embodiments, deployment 214 can accept various requests through interface 211 to configure and deploy local execution environment configuration 402.
[0051] For example, as shown at 410, a request to enable (or disable) data flow management for the identified local execution environment can be received. In some embodiments, as shown at 420, a request to upload or modify a destination connector can also be received. For example, executable script code or other information can be provided to define or implement a connector. In some embodiments, an operation to edit or modify an existing destination connector can be performed (e.g., to change the encoding type or utilize other aspects of the connector).
[0052] As described above, deployment 214 can deploy various IoT service applications to a recipient device. For example, a request 430 for deploying a local execution environment (or updating an existing version of a local execution environment) can be received. This request can utilize a local execution environment configuration 402 specified or modified according to requests 410 and 420. In some embodiments, deployment 214 can then send or otherwise provide the local execution environment for deployment on a client device 404 (e.g., an edge device) to be implemented as a data export management 408 for a local execution environment 406, as shown in 440.
[0053] Client device 450 may be a client device connected to and in communication with client device 404 as part of a client network, as described above with respect to Figure 1 as discussed. In some embodiments, such communications may include one or more requests to create, update, delete, and / or otherwise specify 460 an export configuration for a corresponding data stream through the data export management interface 452 (e.g., according to one or more APIs). A variety of different features, options, parameters, or other criteria can be specified for a data stream (e.g., identified by an identifier, source (e.g., client device, etc.)). For example, a destination configuration may include a target service or other data stream destination, a structure, file, endpoint, or object within this service for receiving the data stream, data formatting (e.g., the type of connector to be used), and / or credentials or other information for accessing the destination through export data management. In some embodiments, the configuration request 410 may specify a priority value for the data stream (e.g., by a numeric value or a label). In some embodiments, priority processing may be specific to certain conditions (e.g., after a reconnect) or may be applied / considered in the absence of specific network conditions.
[0054] In some embodiments, the configuration request 410 may specify network connection criteria. For example, the minimum bandwidth (e.g., to ensure high quality data transmission or speed), link type (e.g., wireless, wired, satellite, etc.) and / or link cost for exporting data stream data, or other network connection conditions that may affect data stream selection behavior may be specified. In some embodiments, the data stream configuration request 410 may specify whether to enable or disable batch processing of data streams (e.g., batching a certain number of records, records of a certain size together, and / or the amount of time since a batch of records was sent together). In some embodiments, export storage configuration information (e.g., format, additional data to be added, whether data can be overwritten, etc.) may be specified as part of the data stream export configuration request 410. In some embodiments, the configuration specified in the request 410 may be included in the data export management feature of the local execution environment deployed using the local execution environment configuration 402.
[0055] Separate from the request for specifying the export configuration of the data stream, client device 450 may send data stream data to client device 404, as shown at 462, and this data stream data may be exported by data export management 408. For example, a separate interface or protocol for transmitting the data stream to client device 404 (rather than interface 452) may be utilized.
[0056] Although described and illustrated in the context of a provider network implementing an IoT service Figures 2 to 4 ,but Figures 2 to 4The various components shown and described in the can be easily applied to other edge networks that export data to other systems, networks or services. Figures 2 to 4 Other embodiments of managed data export from edge devices to provider networks are not intended to be limited. Figure 5 A high-level flow chart of various methods and techniques for implementing managed data export from an edge device to a provider network according to some embodiments is shown. Various different systems and devices can implement the various methods and techniques described below individually or in collaboration. For example, a local execution environment that implements the above-described data export management can use some or all of the various methods. Therefore, the examples above and / or any other systems or devices referred to as performing the described methods are not intended to be limited to other different components, modules, systems or devices.
[0057] In various embodiments, a network connection can be established between an edge device and a remote network. For example, various protocols, credential exchanges, and / or other communications can be performed to allow communications to be sent from an edge device to a target service or another data flow destination in a remote network. When a network connection becomes available or fails, operation of a client device in a client network can continue by sending a data flow to an edge device for later export to a target service or another data flow destination - even when the data flow cannot be executed when the network connection is unavailable (e.g., offline). However, in some embodiments, as shown in 510, a network connection between an edge device and a target service or another data flow destination in a provider network can be reestablished. In this way, data can be exported from various data flows again.
[0058] In some embodiments, as shown in 520, based on the export configuration, the edge device can select one of the different data streams received at the edge device from different devices in the client network to the computing device. For example, a selection technique can be performed to identify and select data records, blocks, or other portions that have not yet been sent (e.g., identifiable by a checksum, watermark, or export status of the data stream) to select the highest priority data stream. In some embodiments, the selection technique applied after the network connection is reestablished can be different from the selection technique applied before the network connection becomes unavailable. As described above with respect to Figure 3 As discussed, various other analyses or criteria may be specified in the export configuration, such as LIFO or other scheduling techniques, which may be applied alone or in combination.
[0059] In some embodiments, the edge device can identify the next portion of data from the selected data stream to be sent to the target service or another data stream destination, as shown in 530. For example, a checksum or other indicator from the last sent portion of the data stream can be evaluated. In some embodiments, the next portion can be the next portion of this data stream in a FIFO queue, or other order-preserving structures (which can be removed when exporting) can be used to identify the next portion of data from the selected data stream.
[0060] In some embodiments, as shown in 540, the next portion of the data can be sent by the edge device to a target service or another data stream destination in the remote network. For example, a portion of the data can be encoded, compressed, encrypted, or otherwise formatted according to the requirements of the target service or another data stream destination, such as, converted from one type of data format to another type of data format. One or more requests formatted according to an interface (e.g., an API) for the target service or another data stream destination can be generated, filled with the next portion of the data, and sent. In some embodiments, a confirmation or another response can be received to update or mark the completed export of the data (e.g., and remove a portion of the data stream from the queue by updating the checksum). In some embodiments, this portion can be a batch of records. In some embodiments, the next portion can not be sent until a minimum batch size of records is received for this data stream.
[0061] The next data stream to be transmitted may continue to be selected (as indicated by the arrow returning to element 520. In this manner, each selection provides the edge device with an opportunity to dynamically adjust which data is sent in response to changing conditions (e.g., with respect to submitted data streams or network connection conditions).
[0062] After the initial selection of a data stream, different types of additional evaluations or criteria may be applied.In various embodiments, network connection criteria may be used to optimize the use of a network connection based on the state of the network connection before sending data. Figure 6 A high-level flow diagram is shown for implementing various methods and techniques for evaluating network connection criteria to derive data from a data stream at a local execution environment in accordance with some embodiments.
[0063] In some embodiments, as shown in 610, the edge device can select a data flow to be sent from the edge device to a target service or another data flow destination in a remote network. For example, the above description of Figure 1 , Figure 3 , Figure 4 as well as Figure 5In various embodiments, an evaluation of network connectivity criteria for the data stream may be performed, as shown at 620. The network connectivity criteria may be included as part of the export configuration and specify various network connectivity states that allow or deny export of data for the data stream.
[0064] For example, the network connection standard can specify the minimum available capacity or bandwidth (e.g., at least 10 Mbps) of the network connection, and if unavailable, refuse to send data from the data stream. In some embodiments, in order to evaluate the network connection standard, the bandwidth value can be estimated according to the historical flux and load of the data stream, and compared with the minimum available value. In some embodiments, the link cost can be defined or specified for certain types of network connections, so that the network connection with the link cost exceeding the specified link cost may not meet the network connection standard, such as, if the capacity is low or the time cost is high (e.g., satellite data communication link and wired data communication link) to run, the link cost is high. In some embodiments, if the network connection standard is met, as shown in 630, the data from the data stream can be sent to the target service or another data stream destination. If not, another data stream can be selected to send, as shown in 640 (e.g., according to the selection technology at element 610).
[0065] In some embodiments, admitting, accepting, and / or storing data received from a data stream may be configured using an export configuration for the data stream. For example, some export configurations may specify whether data stream values may be overridden. In some embodiments, admitting data at a faster rate than a guaranteed or agreed-upon throughput capacity (as specified in the export configuration) may result in throttling or other responsive actions to reduce throughput to the agreed-upon capacity. Figure 7 A high-level flow diagram is shown for implementing various methods and techniques for storing data received from a data flow at an edge device for export to a target service or another data flow destination, in accordance with some embodiments.
[0066] As shown at 710, data can be received from a data stream at a computing device in a client network of a local execution environment hosting a remote network for export to a target service or another data stream destination in the remote network. As shown at 720, a utilization threshold can be applied to determine whether export storage for the data stream (and other data streams) at the computing device is above the utilization threshold. For example, if the amount of used capacity exceeds a threshold of 90% of available capacity, further consideration may be required before storing data for the data stream. If not, as indicated by a negative exit from 720, the data to be exported to the target service or another data stream destination can be stored.
[0067] In some embodiments, as shown at 730, the priority of the data stream can be evaluated based on the export configuration of the data stream. For example, the export configuration can specify a high priority or low priority value for the data stream. If it is determined that the priority value is above a threshold amount or standard, then the priority of storing the data may exist, as indicated by the affirmative exit from 740. If not, the data can be discarded from the data stream, as shown at 760. In some embodiments, data may overwrite stored data for this data stream instead of discarding the data. For example, data stored at the "head" or "front end" of a stream to be exported at an edge device may be overwritten instead of discarding the data.
[0068] In various embodiments, the methods described herein may be implemented by any combination of hardware and software. For example, in one embodiment, the methods may be implemented on one or more computer systems (e.g., Figure 8 The computer system in the embodiment of the present invention is implemented on or across one or more computer systems, the computer system including one or more processors executing program instructions stored on one or more computer-readable storage media coupled to the processor. The program instructions can implement the functions described herein (e.g., the functions of various servers and other components of the network-based virtual computing resource provider described herein). The various methods shown in the accompanying drawings and described herein represent example embodiments of the method. The order of any method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc.
[0069] Embodiments of managed data export from an edge device to a remote network as described herein may be performed on one or more computer systems, which may interact with various other devices. Figure 8 One such computer system is shown. In various embodiments, computer system 1000 may be any of a variety of types of devices, including, but not limited to, a personal computer system, a desktop computer, a laptop computer, a notebook or netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a camera, a set-top box, a mobile device, a consumer device, a video game console, a handheld video game device, an application server, a storage device, a peripheral device (such as a switch, a modem, a router), or generally any type of computing device, computing node, or electronic device.
[0070] In the illustrated embodiment, the computer system 1000 includes one or more processors 1010 coupled to a system memory 1020 via an input / output (I / O) interface 1030. The computer system 1000 also includes a network interface 1040 coupled to the I / O interface 1030, and one or more input / output devices 1050, such as a cursor control device 1060, a keyboard 1070, and a display 1080. The display 1080 may include a standard computer monitor and / or other display systems, technologies, or devices. In at least some embodiments, the input / output device 1050 may also include a touch or multi-touch enabled device, such as a touchpad or tablet computer, through which a user enters input via a stylus-type device and / or one or more digital inputs. In some embodiments, it is contemplated that a single instance of the computer system 1000 may be used to implement the embodiments, while in other embodiments, multiple such systems or multiple nodes making up the computer system 1000 may host different portions or instances of the embodiments. For example, in one embodiment, some elements may be implemented by one or more nodes of computer system 1000 that are distinct from those nodes that implement other elements.
[0071] In various embodiments, computer system 1000 may be a uniprocessor system including one processor 1010, or a multiprocessor system including several processors 1010 (e.g., two, four, eight, or another suitable number). Processor 1010 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1010 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as, for example, an x86, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each of processors 1010 may collectively, but not necessarily, implement the same ISA.
[0072] In some embodiments, at least one processor 1010 may be a graphics processing unit. A graphics processing unit or GPU may be considered as a dedicated graphics rendering device for a personal computer, workstation, game console, or other computing or electronic device. Modern GPUs can be very efficient in manipulating and displaying computer graphics, and their highly parallel structure can make them more efficient than a typical CPU for a series of complex graphics algorithms. For example, a graphics processor may implement a certain number of graphics primitive operations in a manner that performs them faster than drawing directly to the screen with a master central processing unit (CPU). In various embodiments, graphics rendering may be implemented at least in part by program instructions executed on one of such GPUs or executed in parallel on two or more of such GPUs. A GPU may implement one or more application programming interfaces (APIs) that allow programmers to call the functions of a GPU. Suitable GPUs may be commercially available from suppliers such as NVIDIA Corporation and ATI Technologies (AMD).
[0073] The system memory 1020 may store program instructions and / or data that may be accessed by the processor 1010. In various embodiments, the system memory 1020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data that implement the desired functions such as those described above are shown as being stored in the system memory 1020 as program instructions 1025 and data memory 1035, respectively. In other embodiments, program instructions and / or data may be received, sent, or stored on different types of computer accessible media or on similar media separate from the system memory 1020 or the computer system 1000. In general, non-transitory computer-readable storage media may include storage media or memory media (such as magnetic or optical media), for example, a disk or CD / DVD-ROM coupled to the computer system 1000 via the I / O interface 1030. Program instructions and data stored by computer-readable media can be transmitted by transmission media or signals (such as electrical signals, electromagnetic signals or digital signals), which can be conveyed through communication media (such as networks and / or wireless links), such as can be implemented through network interface 1040.
[0074] In one embodiment, I / O interface 1030 can coordinate I / O traffic between processor 1010, system memory 1020, and any peripheral devices in the device, including network interface 1040 or other peripheral interfaces, such as input / output device 1050. In some embodiments, I / O interface 1030 can perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory 1020) into a format suitable for use by another component (e.g., processor 1010). In some embodiments, I / O interface 1030 can include support for devices attached through various types of peripheral buses, such as, for example, a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, the functionality of I / O interface 1030 can be divided into two or more separate components, such as, for example, a north bridge and a south bridge. In addition, in some embodiments, some or all of the functionality of I / O interface 1030, such as an interface to system memory 1020, can be directly incorporated into processor 1010.
[0075] The network interface 1040 may allow data to be exchanged between the computer system 1000 and other devices attached to the network (such as other computer systems) or between nodes of the computer system 1000. In various embodiments, the network interface 1040 may support communications over a wired or wireless general purpose data network such as, for example, any suitable type of Ethernet; over a telecommunications / telephone network such as an analog voice network or a digital fiber optic communications network; over a storage area network such as a Fibre Channel SAN, or over any other suitable type of network and / or protocol.
[0076] In some embodiments, input / output devices 1050 may include one or more display terminals, keyboards, keypads, touch pads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or retrieving data by one or more computer systems 1000. Multiple input / output devices 1050 may be present in computer system 1000, or may be distributed across various nodes of computer system 1000. In some embodiments, similar input / output devices may be separate from computer system 1000 and may interact with one or more nodes of computer system 1000 via a wired or wireless connection (such as, via network interface 1040).
[0077] like Figure 8As shown, memory 1020 may include program instructions 1025 that implement various methods and techniques as described herein, and data storage 1035 that includes various data that can be accessed by program instructions 1025. In one embodiment, program instructions 1025 may include software elements of embodiments as described herein and as shown in the accompanying drawings. Data storage 1035 may include data that can be used in embodiments. In other embodiments, other or different software elements and data may be included.
[0078] It will be appreciated by those skilled in the art that computer system 1000 is only illustrative and is not intended to limit the scope of the technology described herein. Specifically, computer systems and devices may include any combination of hardware or software that can perform the indicated functions, including computers, personal computer systems, desktop computers, laptop computers, notebook or netbook computers, large computer systems, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, network devices, Internet appliances, PDAs, wireless phones, pagers, consumer devices, video game consoles, handheld video game devices, application servers, storage devices, peripheral devices (such as switches, modems, routers) or generally any type of computing or electronic equipment. Computer system 1000 may also be connected to other devices not shown, or alternatively, may be run as an independent system. In addition, in some embodiments, the functionality provided by the components shown may be combined in fewer components or distributed in additional components. Similarly, in some embodiments, the functions of some components in the components shown may not be provided, and / or other additional functions may be used.
[0079] It will also be understood by those skilled in the art that, although various items are described as being stored in memory or on storage when in use, for the purpose of memory management and data integrity, these items or portions thereof may be transferred between memory and other storage devices. Alternatively, in other embodiments, some or all of the software components may be executed in memory on another device and communicated with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or portable article to be read by an appropriate drive, various examples of which have been described above. In some embodiments, instructions stored on a non-transitory computer-accessible medium separated from the computer system 1000 may be transmitted to the computer system 1000 via a transmission medium or a signal (e.g., an electrical signal, an electromagnetic signal, or a digital signal) conveyed by a communication medium (e.g., a network and / or a wireless link). Various embodiments may also include receiving, sending, or storing instructions and / or data implemented according to the foregoing description on a computer-accessible medium. Therefore, the present invention may be implemented with other computer system configurations.
[0080] It should be noted that any one of the distributed system embodiments described herein or any one of their parts can be implemented as one or more web services. In certain embodiments, network-based services can be implemented by software and / or hardware systems that are designed to support the interoperable machine-to-machine interaction through the network. Network-based services can have interfaces described in machine-processable formats (such as, Web Services Description Language (WSDL)). Other systems can interact with web services in a manner specified by the description of the interface of the network-based service. For example, network-based services can describe the various operations that other systems can call, and can describe specific application programming interfaces (APIs), and when requesting various operations, it can be expected that other systems meet the specific application programming interfaces.
[0081] In various embodiments, a network-based service may be requested or invoked using a message that includes parameters and / or data associated with a network-based service request. This message may be formatted according to a specific markup language such as Extensible Markup Language (XML), and / or may be encapsulated using a protocol such as Simple Object Access Protocol (SOAP). To perform a web service request, a network-based service client may assemble a message including the request using an Internet-based application layer transport protocol such as Hypertext Transfer Protocol (HTTP), and communicate the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the web service.
[0082] In some embodiments, web services can be implemented using representational state transfer ("RESTful") techniques rather than message-based techniques. For example, a web service implemented according to RESTful techniques can be called via parameters contained within an HTTP method (such as PUT, GET, or DELETE) rather than being encapsulated within a SOAP message.
[0083] Embodiments of the present invention may be described in terms of the following:
[0084] Clause 1. A system comprising:
[0085] at least one processor; and
[0086] a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement a local execution environment for a provider network implemented within a client network, the local execution environment being configured to:
[0087] receiving a plurality of different data streams from a plurality of different devices in the client network;
[0088] storing the plurality of different data streams;
[0089] reestablishing a network connection between the local execution environment and a target service in the provider network;
[0090] Selecting one of the plurality of data streams to export to the target service according to an export configuration;
[0091] obtaining a next portion of the data stored for the selected data stream to send to the target service; and
[0092] A next portion of the data is sent to the target service in the provider network according to an interface of the target service.
[0093] Clause 2. The system of clause 1, wherein the local execution environment is further configured to:
[0094] Prior to storing data of at least one of the plurality of different data streams, determining that a priority assigned to the at least one data stream allows storing the data when utilization of the derived storage of the plurality of data streams is above a utilization threshold is performed.
[0095] Clause 3. The system of any of clauses 1-2, wherein the local execution environment is further configured to:
[0096] Before sending a next portion of the data to a target service in the provider network, it is determined that network connectivity criteria specified for a selected data flow in the export configuration are satisfied.
[0097] Clause 4. A system of any of clauses 1-3, wherein the target service is an Internet of Things (IoT) service provided by the provider network, wherein the local execution environment is deployed by the IoT service, and wherein the local execution environment is further configured to receive one or more requests from one of the client devices specifying the export configuration.
[0098] Clause 5. A method comprising the steps of:
[0099] After network connectivity is reestablished between the edge device in the client network and the data flow destination in the remote network:
[0100] selecting, by the local execution environment, one of a plurality of different data streams received at the edge device from a plurality of devices in a client network connected to the edge device based on the export configuration;
[0101] identifying, by the edge device, a next portion of data from the selected data flow to be sent to the data flow destination; and
[0102] A next portion of the data is sent by the edge device from the edge device to a data flow destination in the remote network.
[0103] Clause 6. The method of clause 5, further comprising the following steps:
[0104] Before selecting the one data stream:
[0105] selecting, by the edge device, another data flow to be sent to the data flow destination; and
[0106] determining, by the edge device, that a network connection criterion specified for the selected other data flow in the export configuration of the selected other data flow is not satisfied; and
[0107] Wherein, the selecting the one data stream is performed in response to determining that a network connection criterion specified for the selected other data stream in the export configuration of the selected another data stream is not satisfied.
[0108] Clause 7. The method of clause 6, wherein determining that a network connection criterion specified for the selected other data flow in the export configuration for the selected other data flow is not satisfied comprises evaluating a link cost of the network connection.
[0109] Clause 8. The method of any of clauses 5-7, wherein the export configuration specifies a priority for the selected one data stream, and wherein selecting the one data stream comprises comparing the priorities specified for the multiple different data streams to select the data stream having the highest priority.
[0110] Clause 9. The method of any of clauses 5-8, wherein sending the next portion of the data from the computing device to the data stream destination comprises sending the next portion as part of a batch of a portion of data from the selected data stream.
[0111] Clause 10. The method of any one of clauses 5 to 9, further comprising the following steps:
[0112] The priority assigned by the edge device to at least one of the plurality of data flows allows data of the at least one data flow to be stored at the edge device for subsequent export when utilization of export storage for the plurality of data flows is above a utilization threshold.
[0113] Clause 11. The method of any one of clauses 5 to 9, further comprising the following steps:
[0114] The priority assigned to at least one of the plurality of data flows determined by the local execution environment does not allow data of the at least one data flow to be stored at the edge device for subsequent export when utilization of export storage for the plurality of data flows is above a utilization threshold.
[0115] Clause 12. The method of any one of clauses 5 to 11, further comprising the following steps:
[0116] One or more requests specifying the export configuration are received at the edge device from one of the client devices.
[0117] Clause 13. The method of clause 12, wherein sending the next portion of the data from the edge device to the data flow destination comprises formatting the next portion of the data according to an interface for the data flow destination.
[0118] Clause 14. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement an edge device that implements:
[0119] reestablishing a network connection between the edge device and a data flow destination in a remote network;
[0120] selecting, based on the export configuration, one of a plurality of data streams received from a plurality of devices in a client network including the edge device;
[0121] obtaining a next portion of data to be sent to the data stream destination from the selected data stream received by the edge device; and
[0122] A next portion of the data is sent to the data flow destination in the remote network according to the interface for the data flow destination.
[0123] Clause 15. The one or more non-transitory computer-readable storage media of clause 14, wherein the one or more non-transitory computer-readable storage media store additional program instructions that, when executed on or across the one or more computing devices, cause the edge device to further implement:
[0124] When utilization of the export storage of the plurality of data streams is above a utilization threshold, determining that the priority assigned to at least one of the plurality of data streams allows storage of data of the at least one data stream for subsequent export.
[0125] Clause 16. The one or more non-transitory computer-readable storage media of any of clauses 14-15, wherein the one or more non-transitory computer-readable storage media store additional program instructions that, when executed on or across the one or more computing devices, cause the edge device to further implement:
[0126] Before sending a next portion of the data to a target service in the provider network, it is determined that network connectivity criteria specified for the selected data flow in the export configuration are satisfied.
[0127] Clause 17. One or more non-transitory computer-readable storage media of clause 16, wherein, upon determining that network connection criteria specified for a selected data flow in the export configuration are met, the edge device implements determining that a minimum bandwidth for sending a next portion of the data over the network connection is available.
[0128] Clause 18. The one or more non-transitory computer-readable storage media of any of clauses 14-17, wherein the export configuration specifies a data flow destination of the remote network.
[0129] Clause 19. One or more non-transitory computer-readable storage media of any of clauses 14-18, wherein, when a next portion of the data is sent to the data stream destination, the program instructions cause the edge device to implement sending the next portion as part of a batch of data from the selected data stream.
[0130] Clause 20. One or more non-transitory computer-readable storage media of any of clauses 14-19, wherein sending the next portion of the data to the data flow destination is performed by a connector deployed from an Internet of Things (IoT) service to the edge device as part of a local execution environment, and wherein the connector is a user-submitted connector to the IoT service.
[0131] The various methods shown in the drawings and described herein represent example embodiments of the methods. The methods may be implemented in software, hardware, or a combination thereof. The order of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.
[0132] Various modifications and changes may be made which will be apparent to those skilled in the art having the benefit of this disclosure. The invention is intended to embrace all such modifications and changes, and, therefore, the above description is to be regarded as illustrative rather than restrictive.
Claims
1. A system comprising: at least one processor; and a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement a local execution environment of a provider network implemented within a client network, the local execution environment being configured to: receiving a plurality of different data streams from a plurality of different devices in the client network; storing the plurality of different data streams; reestablishing a network connection between the local execution environment and a target service in the provider network; After re-establishing the network connection, selecting one of the plurality of data streams according to the export configuration to export to the target service; obtaining a next portion of the data stored for the selected data stream to send to the target service; as well as A next portion of the data is sent to the target service in the provider network according to an interface of the target service.
2. The system according to claim 1, wherein: The local execution environment is further configured to: Prior to storing data of at least one of the plurality of different data streams, determining that a priority assigned to the at least one data stream allows storing the data when utilization of the derived storage of the plurality of data streams is above a utilization threshold is performed.
3. The system according to claim 1, wherein: The local execution environment is further configured to: Before sending a next portion of the data to a target service in the provider network, it is determined that network connectivity criteria specified for the selected data flow in the export configuration are satisfied.
4. The system according to claim 1, wherein: The target service is an IoT service provided by the provider network, wherein the local execution environment is deployed by the IoT service, and wherein the local execution environment is further configured to receive one or more requests from one of the client devices specifying the export configuration.
5. A method comprising the following steps: After network connectivity is reestablished between the edge device in the client network and the data flow destination in the remote network: selecting, by the local execution environment, one of a plurality of different data streams received at the edge device from a plurality of devices in a client network connected to the edge device based on the export configuration; identifying, by the edge device, a next portion of data from the selected data flow to be sent to the data flow destination; as well as A next portion of the data is sent by the edge device from the edge device to a data flow destination in the remote network.
6. The method according to claim 5, further comprising the steps of: Before selecting the one data stream: selecting, by the edge device, another data flow to be sent to the data flow destination; and determining, by the edge device, that a network connection criterion specified for the selected other data flow in the export configuration of the selected other data flow is not satisfied; and Wherein, the selecting the one data stream is performed in response to determining that a network connection criterion specified for the selected other data stream in the export configuration of the selected other data stream is not satisfied.
7. The method according to claim 6, wherein: Determining that a network connection criterion specified for the selected another data flow in the export configuration for the selected another data flow is not met includes evaluating a link cost of the network connection.
8. The method according to claim 5, wherein: The export configuration assigns a priority to the selected one data stream, and wherein selecting the one data stream includes comparing the priorities assigned to the plurality of different data streams to select the data stream having the highest priority.
9. The method according to claim 5, wherein: Sending the next portion of the data from the computing device to the data stream destination includes sending the next portion as part of a batch of a portion of data from the selected data stream.
10. The method according to claim 5, further comprising the steps of: The priority assigned by the edge device to at least one of the plurality of data flows allows data of the at least one data flow to be stored at the edge device for subsequent export when utilization of export storage for the plurality of data flows is above a utilization threshold.
11. The method according to claim 5, further comprising the steps of: The priority assigned to at least one of the plurality of data flows determined by the local execution environment does not allow data of the at least one data flow to be stored at the edge device for subsequent export when utilization of export storage for the plurality of data flows is above a utilization threshold.
12. The method according to claim 5, further comprising the steps of: One or more requests specifying the export configuration are received at the edge device from one of the client devices.
13. The method according to claim 12, wherein: Sending the next portion of the data from the edge device to the data flow destination includes formatting the next portion of the data according to an interface for the data flow destination.
14. A non-transitory computer-readable storage medium storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement an edge device that implements: reestablishing a network connection between the edge device and a data flow destination in a remote network; After reestablishing the network connection, selecting, based on the export configuration, one of a plurality of data streams received from a plurality of devices in a client network including the edge device; obtaining a next portion of data to be sent to a destination of the data stream from the selected data stream received by the edge device; as well as A next portion of the data is sent to the data flow destination in the remote network according to the interface for the data flow destination.
15. The non-transitory computer-readable storage medium of claim 14, wherein: The one or more non-transitory computer-readable storage media store additional program instructions that, when executed on or across the one or more computing devices, cause the edge device to further implement: When utilization of the export storage of the plurality of data streams is above a utilization threshold, determining that the priority assigned to at least one of the plurality of data streams allows storage of data of the at least one data stream for later export.
