Method and system for data pipeline stratification for low latency access

Abstract

To provide a method and distributed system for data pipeline stratification for low latency access.SOLUTION: A method at a data pipeline in a computing system includes: receiving data and a schema identifier associated with the data; confirming that the schema identifier is valid; and routing the data at the data pipeline based on the schema identifier. In addition, the computing system has a processor, a data pipeline, and a communications subsystem. The computing system is configured to: receive data and a schema identifier associated with the data; confirm, using the processor, that the schema identifier is valid; and route the data at the data pipeline based on the schema identifier.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to distributed systems, and more particularly to data pipelines in distributed systems. [Background technology]

[0002] background In a data processing ecosystem, data may enter the system and may need to be processed in various ways. However, some data may require processing that can cause delays in the system.

[0003] In one example, the data processing ecosystem may relate to a vehicle. Modern vehicles have different processors, control units, sensors, and other computing nodes. Such sensors may be distributed among various computing nodes on the vehicle, and each computing node may have access to zero, one, or more sensor drivers. A sensor or group of sensors may be used to generate information that may be useful for one or more applications. Such information is referred to herein as insights.

[0004] Software can be developed to leverage such insights, however, accessing such data may require rapid transfer of data from a cloud VM or external data network containing the vehicle data to internal storage so that the data can be retrieved within a short period of time. Summary of the Invention [Means for solving the problem]

[0005] The present disclosure provides a method in a data pipeline in a computing system, the method including receiving data and a schema identifier associated with the data, verifying that the schema identifier is valid, and routing the data in the data pipeline based on the schema identifier.

[0006] The present disclosure further provides a computing system including a processor, a data pipeline, and a communication subsystem, the computing system configured to receive data and a schema identifier associated with the data, verify, using the processor, that the schema identifier is valid, and route the data in the data pipeline based on the schema identifier.

[0007] The present disclosure further provides a computer-readable medium for storing instruction code that, when executed by a processor in a computing system, causes the computing system to receive data and a schema identifier associated with the data, verify, using the processor, that the schema identifier is valid, and route the data in a data pipeline based on the schema identifier.

[0008] In many data-driven systems, a fast data pipeline that facilitates efficient and fast transfer of data from cloud virtual machines (VMs) or external data networks may be required. Such data may, for example, in some cases be transferred to internal storage on a computing device so that the data can be read within a short period of time by parties in the system who wish to access this data externally. For such pipelines, low latency may be important where data transfer efficiency may be required.

[0009] In this regard, according to embodiments of the present disclosure, a generic data pipeline was designed to be efficient in storing data in its original form with little processing other than validation against a schema, and further processing was designed with parallelism to achieve the same or similar latency between input of the original data and subsequent access of the generated data.

[0010] According to an embodiment of the present disclosure, the data base layer may consist of a generic data pipeline that will initially receive data. The data may be in a predefined format, provided that a schema for that format can be pre-registered. For example, the data may initially be in JavaScript® Object Notation (JSON) format, with a chosen schema being JSONSchema. However, other data formats and schemas are possible, and the present disclosure is not limited to a particular data format and schema.

[0011] In some cases, a simple application program interface (API) may be used to send data to a distributed system. For example, the API may be a data entry REpresentational State Transfer (REST) ​​API. However, other forms of APIs are possible.

[0012] The data may then be stored in a database, which in some embodiments may be a Not Only Structured Query Language (NoSQL) database, although other databases may be used as well.

[0013] The data can then be accessed in the distributed system in the same format using a defined API (e.g., in some cases a data access REST API).

[0014] Schema registration may be available on the input side, and querying the schema may be available on both the input and access APIs.

[0015] In addition to data input APIs, data can be streamed into distributed systems via data ingestion conduits such as Kinesis® in Amazon® Web Services (AWS), Azure® Event Hubs, among others, and via Internet of Things (IoT) core channels, among other options.

[0016] A conduit or channel routes all of the data into a general data pipeline. While this single channel can potentially become a bottleneck for larger data streams, this can be alleviated by creating duplicate lanes for the general data pipeline, similar to express lanes on a highway that start out as one lane and eventually expand to multiple lanes. This represents a baseline parallelization for handling throughput while maintaining low latency.

[0017] Additional parallelization can be further used to achieve low latency for the processed data. While it is not possible to achieve the same level of latency as for the pre-processed data, such parallelization improves system performance. In this regard, the first step is to branch the incoming data along different paths mid-pipeline to ensure that copies of the data are diverted to separate lanes to handle slower processing for archival and storage, and that primary data is diverted to the data bus.

[0018] For example, the slower path may include database storage where replication and connection setup potentially introduce intermittent delays.

[0019] Slower paths may also include archival storage. For example, Simple Storage Service (S3) or Azure Blob, among other storage services, are high-latency operations. While the use of S3 is used below for illustration, this disclosure is not limited to any particular storage.

[0020] Slower paths may also include external streaming. For example, streaming to external channels may include HyperText Transfer Protocol (HTTP) streams, data ingestion conduit streaming, or redirects to third-party storage, among others, all of which may have intermittent high latency.

[0021] Sending primary data to the data bus provides several benefits. In a first aspect, the bus may itself be a data cache that can be queried, for example, via a query language such as GraphQL, or other real-time services, potentially acting as a short-term source for low-latency queries, among other options. In one case, tenants interested in quick queries may potentially hit the data bus for data only if they specify a short-term timestamp range for their query.

[0022] In a second aspect, data processing components can be implemented to subscribe to data on various topics (one topic per schema) and can be triggered to process incoming data when it arrives on a topic of interest. This event-driven logic can result in massive parallelization. Components can be implemented for various types of processing. Such processing can include insight generation (as described below), analytical processing, and data disaggregation to transform data into different organizations to aid in state tracking in a timely manner.

[0023] In a third aspect, any generated / processed data can be fed back to the bus for further downstream processing by components, and simultaneously branched off into slower data lanes for upstream archiving, storage, and external streaming, similar to how data was previously branched off in flow from the general-purpose data pipeline.

[0024] The foregoing may be implemented in any distributed data processing system. It is described below with respect to a vehicle system in which data may be processed to create insights for the system. However, this is provided merely as an example system, and the present disclosure is not limited to any particular distributed data processing system.

[0025] Vehicle Systems

[0026] In modern vehicles, information from one or more physical sensors can be processed to create potentially valuable "insights" in the system. Such one or more physical sensors and their associated processing can be logically referred to as a microservice or synthetic sensor (SS). The terms microservice and synthetic sensor are used interchangeably herein.

[0027] Synthetic sensors may exist in other types of applications, including but not limited to medical applications, manufacturing applications, Internet of Things applications, among others, and the present disclosure is not limited to vehicle applications, a vehicle application being provided below for illustrative purposes.

[0028] Insight is a term used herein to describe any computer-generated interpretation of basic sensor data. Insights can be as simple as data aggregation or data correlation, or as complex as artificial intelligence and machine learning. For example, a temperature sensor providing high and low watermarks for notifications can be considered an "insight." For traffic light services, geofencing is an insight. For cameras, occupant recognition can be an insight. For example, the use of a combination of a sensor, such as a temperature sensor, and a camera can be used with an artificial intelligence model to determine if a car seat is occupied in a hot vehicle, which can be an insight. Many other examples of insights are possible.

[0029] In one embodiment, vehicle applications can be implemented in a system that provides consistent access to vehicle data and intelligent insights in a way that is familiar and accessible to the developer community. Such an environment can enable cloud developers to extend their reach to the edge within the vehicle through the development of synthetic sensors that gain intelligent insights about vehicle data using common cloud development techniques and paradigms. Such an environment can provide consistent access to vehicle data such that synthetic sensors can be written and deployed across a wide range of vehicle bases without customization.

[0030] Specifically, although the hardware and software on each vehicle may differ from that of other vehicles, the use of abstractions allows similar data to be provided to the synthetic sensors. Some vehicles may be configured with particular sensors, hardware, electronic control units (ECUs), or operating systems (OSs) that may not be present in other vehicles. The present disclosure provides, for example: (Item 1) 1. A method in a data pipeline in a computing system, the method comprising: receiving data and a schema identifier associated with the data; Verifying that the schema identifier is valid; and Routing data in a data pipeline based on a schema identifier; A method comprising: (Item 2) The method of any preceding item, wherein routing includes routing the data to a second processing function in the data pipeline. (Item 3) Routing is a cloning the data to create cloned data; Transmitting the data to a destination; Sending clone data to the processing function 4. The method of claim 1, wherein the processing function includes a slower path than transmitting the data to the destination. (Item 4) receiving a new schema identifier; determining a data path for data associated with the new schema identifier; The method according to any one of the preceding items, further comprising: (Item 5) 10. The method of claim 1, further comprising creating a new processing function for data associated with the new schema identifier. (Item 6) 10. The method of any of the preceding items, wherein a parallel data pipeline is created when the amount of data reaches a threshold. (Item 7) 10. The method of claim 1, wherein when the amount of data reaches a threshold, a second processing function is replicated in the data pipeline. (Item 8) Item 11. The method of any of the preceding items, wherein verifying that the schema identifier is valid includes checking that the schema of the data matches the schema of the schema identifier. (Item 9) 10. The method of claim 1, wherein routing the data includes routing the data to a bus that includes a real-time data cache. (Item 10) a processor; A data pipeline and Communication subsystem and 1. A computing system comprising: receiving data and a schema identifier associated with the data; using a processor to verify that the schema identifier is valid; Routing data in a data pipeline based on a schema identifier; 1. A computing system configured to: (Item 11) 2. The computing system of claim 1, wherein the computing system is configured to route the data by routing the data to a second processing function in the data pipeline. (Item 12) The computing system cloning the data to create cloned data; Transmitting the data to a destination; Sending clone data to the processing function 4. The computing system of claim 1, wherein the processing function includes a slower path than the one used to send the data to the destination. (Item 13) The computing system receiving a new schema identifier; determining a data path for data associated with the new schema identifier; 2. The computing system of claim 1, further configured to: (Item 14) 2. The computing system of claim 1, wherein the computing system is further configured to create a new processing function for data associated with the new schema identifier. (Item 15) 10. The computing system of claim 1, wherein a parallel data pipeline is created when the amount of data reaches a threshold. (Item 16) 2. The computing system of claim 1, wherein when the amount of data reaches a threshold, the second processing function is replicated in the data pipeline. (Item 17) 10. The computing system of claim 1, wherein the computing system is configured to verify that the schema identifier is valid by checking that the schema of the data matches the schema of the schema identifier. (Item 18) 10. The computing system of claim 1, wherein the computing system is configured to route data by routing the data to a bus that includes a real-time data cache. (Item 19) A computer-readable medium for storing instruction code, the instruction code, when executed by a processor in a computing system, receiving data and a schema identifier associated with the data; using a processor to verify that the schema identifier is valid; Routing data in a data pipeline based on a schema identifier; A computer-readable medium for causing a computing system to perform the following: (Summary) A method and system for data pipeline layering for low-latency access is described. The method includes receiving data and a schema identifier associated with the data in a data pipeline within a computing system, verifying that the schema identifier is valid, and routing the data in the data pipeline based on the schema identifier. The computing system also includes a processor, a data pipeline, and a communications subsystem, and is configured to receive the data and the schema identifier associated with the data, verify using the processor that the schema identifier is valid, and route the data in the data pipeline based on the schema identifier. [Brief explanation of the drawings]

[0031] The present disclosure may be better understood with reference to the drawings.

[0032] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary computing node in a computer system.

[0033] [Figure 2] FIG. 2 is a block diagram illustrating a system for use with embodiments of the present disclosure.

[0034] [Figure 3] FIG. 3 is a block diagram illustrating an exemplary system having a low-latency data processing pipeline.

[0035] [Figure 4] FIG. 4 is a block diagram illustrating another exemplary system having a low-latency data processing pipeline.

[0036] [Figure 5]FIG. 5 is a block diagram illustrating an exemplary data processing pipeline.

[0037] [Figure 6] Figure 6 is a process diagram illustrating the process of creating and using processing functions within a data processing pipeline.

[0038] [Figure 7] FIG. 7 is a block diagram of a simplified computing device that can be used with embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0039] For example, reference is now made to Figure 1, which illustrates a node 110 in a vehicle. A node, as used herein, may be one or a group of an electronic control unit, a central processing unit, or a kernel control, among other options, and may be considered a single control unit.

[0040] 1, node 110 includes a service manager 120 that can interact with drivers for sensors to which the node is connected. For example, node 110 may have access to a location sensor, such as a Global Positioning System (GPS) chipset, as shown in block 122.

[0041] To enable node 110 to interact with modules on other nodes and to provide functionality to the computing system, a hardware abstraction layer (HAL) may be provided on node 110, including HAL services 130. Each HAL service 130 is responsible for sensor integration and may provide various functions, including integrating into underlying sensors, normalizing sensor data, and / or providing a barrier between certified and uncertified software, if required. Other functions for HAL services are also possible.

[0042] In the example of FIG. 1, the HAL is provided for camera information, as indicated using block 132 .

[0043] 1 shows node 110 with a single service and a single HAL, this is provided for illustrative purposes only. Node 110 may have a single service with no HAL, a single HAL with no service, multiple HALs with no HALs, multiple HALs with no service, and / or a combination of services and HALs.

[0044] One example of a system that may use node 110 would be an application development environment for a vehicle. Such an application development environment may develop, among other things, applications for user experience including comfort, navigation, and entertainment information, applications for safety, applications for fleet management, applications for performance monitoring, or other such applications for the vehicle environment. In particular, vehicles offer multiple sensors, and different makes, models, or brands may use different sensors with different data formats or values, creating fragmented sensor readings that depend on such sensors. This fragmentation hinders the development of an application ecosystem that utilizes vehicle data. Additionally, low-level sensor data is often too granular to be easily useful for applications.

[0045] In this regard, a hardware abstraction layer can be used to provide a hardware-independent sensor interface / abstraction that can encapsulate interactions with underlying sensor drivers. The use of hardware abstractions in various computing nodes creates a scalable platform that can, for example, provide barriers between modules for implementing safety-certified systems that are isolated from other systems, among other options.

[0046] Applications do not directly interact with the sensor hardware to access sensor data; instead, they leverage the Hardware Abstraction Layer. This separation provides a clear distinction between the responsibilities of the HAL (sensor integration and sensor data normalization) and other abstractions, such as the Vehicle Abstraction Layer (VAL), which is used to manage access to vehicle data and provide value-added insights.

[0047] Specifically, insights may leverage sensor data from multiple HALs via the VAL to provide vehicle abstraction and value-added insights. The VAL may control access to a normalized form of vehicle data, while the synthetic sensor may provide value-added inferences derived from data obtained from the VAL. Examples may include a signal service that may provide consistently formatted location coordinate data from which the synthetic sensor may calculate insights such as geofencing. The VAL may provide access to battery signals that may be used by the synthetic sensor to calculate charge state / consumption / expected time remaining / expected range, or access to vehicle doors and door status, a camera service that may provide video frames from an in-cabin camera to a synthetic sensor that may recognize the driver or passenger and analyze the frames to generate insights, among other things.

[0048] Insight services can leverage sensor data from multiple HALs to provide vehicle abstraction and value-added insights. Higher-level insights into data enable application developers to create future automotive experiences. Insight is a term used to describe any value-added interpretation of basic sensor data. Insights can be as simple as data aggregation or data correlation or as complex as artificial intelligence and machine learning. For example, for a temperature sensor, providing high and low watermarks for notifications can be considered an "insight." For location services, geofencing can be an insight, and for cameras, occupant recognition can be considered an insight.

[0049] 1 may be required for such an application development environment. In many systems, node 110 may need to communicate with other nodes on the vehicle, such as other ECUs, CPUs, or computing systems, which may use a different operating system than node 110.

[0050] Reference is now made to Figure 2, which illustrates an exemplary deployed system showing various generators and consumers of insights. The embodiment of Figure 2 is provided merely for illustrative purposes; in some cases, there will be fewer participants in the system. In other cases, there will be more participants in the system.

[0051] 2, vehicle 200 may include a computing system and a communication system. Portions of the computing system may include regions 210 that may have applications that consume insights, as described below. Additionally, portions of the computing system on vehicle 210 may include edge regions 212. In some embodiments, edge regions 212 may generate insights. However, in other cases, insights may be generated within region 210 or consumed within edge regions 212.

[0052] In the example of Figure 2, vehicle 200 communicates using a communication system with access points (e.g., cellular base stations), shown in Figure 2 as eNB 220. The base stations may communicate with core network 230, which may then forward the communications to cloud service provider 240 through network 232. For example, network 232 may be a wide area network such as the Internet.

[0053] In other embodiments, any technology associated with a particular cellular system or wireless communication protocol, rather than core network 230, may be used with the example vehicle of Figure 2. In other cases, any wired or wireless communication protocol may be used to connect to cloud service 240 or cloud region 250.

[0054] In some embodiments, cloud services 240 may provide developer software development kits (SDKs), edge node interfaces, among other services.

[0055] In some embodiments, cloud domain 250 may generate or consume insights. Cloud domain 250 may communicate with cloud service provider 240 through network 232 and, in some cases, with other domains, such as edge domain 212 on vehicle 200.

[0056] Furthermore, devices 260, rather than vehicles, may consume the insights. Devices 260 may be any computing device capable of generating or consuming such insights and may include Internet of Things devices, mobile devices, medical equipment, vehicles, or equipment related to vehicles, among other options. Devices 260 may communicate over network 232 using various wired or wireless technologies, including, but not limited to, Ethernet, fiber, cellular, Wi-Fi, and satellite, among other options.

[0057] Device 260, in some embodiments, may include region 262 that may consume insights. Additionally, device 260 may, in some cases, include edge region 264 that may generate insights. However, in other cases, region 262 may generate insights and edge region 264 may consume the insights.

[0058] Additionally, while the embodiment of FIG. 2 shows only two regions within vehicle 200 or device 260, in practice only one or multiple regions may exist within vehicle 200 or device 260, and the present disclosure is not limited to having only two regions within any particular device. In particular, device 260 may be used solely to generate insights if it has only a single region. In other cases, device 260 may solely consume insights and similarly have only one region. In other cases, device 260 or vehicle 200 may have multiple regions along with edge region 212.

[0059] In some embodiments, device 260 may be a developer device for the creation of insights for the system.

[0060] Each region or edge region may have an OS-independent architecture. Specifically, the regions shown in FIG. 2 may be implemented on various regions, and different regions may have different operating systems. For example, in a vehicle system, different vehicle manufacturers may use different operating systems. Therefore, the system architecture for the edge region may use an abstraction layer to allow the system to be implemented on different platforms.

[0061] While the example of Figure 2 shows an exemplary system for use with embodiments of the present disclosure, it is provided for illustrative purposes only, and other systems may be similarly used with embodiments herein. Thus, the example of Figure 2 is not limiting.

[0062] Data Pipeline

[0063] A sample architecture for an embodiment of the present disclosure is provided herein using the example of FIG.

[0064] In the example of Figure 3, the system includes at least one edge virtual environment virtual private cloud (VPC) 310, which in some embodiments may be vehicle 200 from the example of Figure 2. Components outside 310 may be in the cloud outside the vehicle, for example, part of cloud services 240 or part of cloud domain 250 from the example of Figure 2.

[0065] In the example of Figure 3, edge virtualization environment VPC 310 includes one or more virtual machines (VMs), shown in the example of Figure 3 as VM 312, VM 314, and VM 316. As will be appreciated by those skilled in the art, the number of VMs depends on the implementation and may be one, two, or more, and the present disclosure is not limited to any particular number of VMs. Furthermore, in some embodiments, a non-virtualized operating system may be used in place of or in addition to VPC 310. Thus, the example virtualization environment of Figure 3 is provided merely for illustrative purposes.

[0066] Each VM may have zero, one, or more synthetic sensors, and in the example of Figure 3, VM 312 includes synthetic sensor 320 and synthetic sensor 322. As shown above, synthetic sensor 320 and synthetic sensor 322 may consume data and provide insights.

[0067] VM 312 is further shown with a hardware abstraction layer 324. Such hardware abstraction layer 324 may receive information from, for example, physical sensors on the vehicle. Such information may be received over a data bus, such as a CANbus, among other options.

[0068] In the test environment, data may come from a simulation data streamer 328, which may receive input from a simulation stream 330 or a command line interface 332. In particular, the VM runtime environment may require the ability for a user to stream or play data into the HAL / VAL layer of the edge virtualization environment VPC 310 to generate input data for synthetic sensors operating in the environment. To do this, a set of local scripts with useful data extracted from available test datasets may be used. The command line interface 332 may be used to request and control the stream of simulated data from the dataset. Additionally, in some cases, the cloud data environment 310 may accept uploaded data and allow third-party developers to stream the uploaded data, select data from a data store, or simulated data controlled by a timeline script that specifies the type of event to simulate.

[0069] Data from various synthetic sensors may be provided to the auxiliary event bridge 326. The auxiliary event bridge 326 may be used to relay incoming data from synthetic sensors on edge devices.

[0070] Thus, data from the synthetic sensors is provided to the data injection layer 333 through the preliminary event bridge 326. In some cases, this communication may go through a firewall 327, which may act as a security layer between the VPC 310 and the data injection layer 333. For example, the firewall 327 may provide access control, traffic filtering, logging, among other similar functionality.

[0071] 3, data injection layer 333 includes one or more of data ingestion conduit 334, IoT core 336, and REST API 338. However, more or fewer data conduits or connections are possible.

[0072] In addition to data from one or more edge virtual environments VPC 310, data may be provided from one or more data bridges. A third-party / original equipment manufacturer (OEM) data bridge 340 is shown in the example of FIG.

[0073] In particular, data bridge 340 can be used to leverage data sources, such as third-party vehicle data brokers, to source real-world vehicle data. This can be done both during testing and prior to deployment of real-world edge devices, and after such deployment for additional data to be provided to the system. Data bridge 340 can be used to extract that data from the OEM and stream it into the system. Thus, data bridge 340 can be comprised of one or more data adapters, which are microservices responsible for independently reading data from a single third-party or OEM data source and uploading the data in bulk into the system, as well as for incrementally polling or receiving streams of data from third-party services, relaying incremental batches of data, and uploading them into the system. Such microservices are shown as OEM / Third-Party Data Sync 342 and OEM / Third-Party Data Sync 344.

[0074] Data sync 342 interacts with data broker 346 and data sync 344 interacts with data broker 348 .

[0075] These two modes for data synchronization can be used for various reasons. First, one function can be to initially upload as much historical data as possible when a new data broker is added. Second, during an initial prototyping / deployment phase, some components of the data pipeline can be refactored and the database built with test data can be purged and redeployed with a new schema that can be repopulated in batches.

[0076] Furthermore, such data may then be available for injection into hosted or local edge VMs. Microservices capable of handling requests for streaming data of notable events may be used for simulated input data for prototyping and / or testing of synthetic sensors.

[0077] Data from the data bridge 340 may be provided to a data injection layer 333 .

[0078] 3, data injection layer 333 may include a data ingestion conduit 334, which in one exemplary implementation may be a Kinesis conduit. In this case, conduit 334 is used for a Kinesis® stream for incoming data coming from devices operating at the edge. Data is received directly via a Kinesis endpoint or redirected from incoming Message Queuing Telemetry Transport (MQTT) messages (IoT Core 336) or incoming REST calls for REST API 338. However, other conduits may be used in data injection layer 333 as well, and this disclosure is not limited to any particular data injection conduit.

[0079] Data from data injection layer 333 is provided to data processing pipeline 350. Specifically, a generic data processing pipeline 350 is provided that can deliver data to a processing function, such as AWS Lambda® or Azure Functions or Azure Automation processing functions, among others. Although the use of Lambda is used below for illustration, reference to Lambda may not imply any processing function.

[0080] The processing pipeline 350 may spawn one or more Lambda functions during processing to perform more advanced validation and post-processing of the incoming data and ultimately route it to the appropriate location. For example, for data routed to S3, this may be done in a manner that partitions the data into separate S3 buckets by schema. In addition to S3 delivery, other options include, for example, Kinesis streams, HTTP streaming, etc., among others. Lambda functions are serverless and event-driven computational services.

[0081] Finally, data may need to be routed internally to other microservices via an internal message broker.

[0082] Thus, the data processing pipeline 350 may include a network of Lambda functions, and a gateway may determine how to route data or clone it to other Lambda functions. However, before large amounts of data with different schemas can be streamed into the pipeline, the data pipeline may need to be sufficiently upgraded to route the data accordingly and to separate data of different schemas from each other.

[0083] After establishing the basic elements in a generic data pipeline, the next step in the process may be to improve S3 bucket organization and delivery of data to S3 so that only one schema of data goes to each bucket. This may make integration with data lakes and cloud machine learning (ML) processing easier, which often expect data in S3 buckets to be consistent.

[0084] For example, data processing pipeline 350 may provide data relay to S3 data storage 352 for storage. Naming conventions for S3 buckets may need to be defined in relation to the schema metadata provided by the schema registration API. Thus, in some cases, the schema registration API may be extended to trigger the creation of a corresponding S3 bucket.

[0085] In some cases, processes can ensure that data delivered to data storage is consistent. For example, if data comes from IoT Core, it is extracted from the IoT message field and then the contents are sent to S3, yet the data may include an MQTT message wrapper. The MQTT message wrapper can be removed before sending the data to S3.

[0086] In some cases, the data processing pipeline 350 can be updated to invoke a new Lambda to route incoming schema-specific validated data to a corresponding S3 bucket. This may require, in some cases, adding a transformation Lambda to the delivery stream to fix the data before going to S3, or possibly adding programmable logic to remove the S3 delivery from the delivery stream and route the data through another custom Lambda function.

[0087] Additionally, in some cases, data may need to be partitioned based on the owner of the data, such as the OEM or a third party. Such data may need to be kept separate or directed to a different set of buckets for each OEM, tenant, or both. Accounting for this may require administrative tracking of output and outgoing buckets, nomenclature for different use cases, permission considerations, among other factors.

[0088] The data processing pipeline 350 may also route data to a streamer 354, which may be used for various streaming services, such as HTTPS streaming, among other options.

[0089] The data processing pipeline 350 may further provide data to a data relay 356, which in some cases may be an OEM or other third party Kinesis.

[0090] In some cases, data processing pipeline 350 may provide data to message broker 360. For example, message broker 360 may be the Apache Kafka® message broker, which is an open-source, topic-based messaging service used for inter-service communication in cloud deployments and is similar to AWS Kinesis. However, other message brokers may be used as well. In this regard, a Kafka cluster may be created as a messaging system that microservices can use to communicate with each other or to route live data that multiple services can consume. For example, an internal Kafka message broker may be used to cache incoming and generated data and insights for generators and processors to consume and share. As with the output of insight data generators, data may be directed from the data processing Lambda (data processing pipeline 350) to the message broker as it is ingested.

[0091] The message broker 360 may provide data to various data processors 362. For example, the data processors 362 may, in some cases, include insight data generators. As data flows in from outside the system, it needs to be stored, where it can be further processed to generate insights / data derived from the incoming data, or it can be normalized and converted into different formats for subsequent analysis. These insights are also stored, for example in a database 364, for future retrieval. The insights can be consumed by other insight data generators operating in the system.

[0092] In some cases, data processor 362 may include a vehicle state generator. As data flows in from outside the system, the data may need to be processed to extract certain characteristics of the vehicle and store a snapshot of certain vehicle characteristics over a time frame (e.g., 60 days). This differs from insight data generators, which operate on an on-demand processing basis. Vehicle state generators may be attempting to crystallize the current state of the vehicle at a given time. The output of these generators may, in some cases, be directed to a different database designed to store rolling data characteristics for a particular vehicle, with data being added and removed to maintain a limited history of the vehicle state. For example, in the embodiment of FIG. 3, this is shown with database 366.

[0093] In some cases, data processor 362 may include an analytics generator. As data flows through the internal data pipeline, whether internally or externally, it may be necessary to analyze this data and store some analytics related to this data from a high-level perspective that may be displayed as a graph or reported in a dashboard. This analytics data may be stored in a separate database, as shown in the example of FIG. 3 as database 368. Additionally, the analytics data may be accessible externally via analytics API 380, as described below. The role of the analytics generator is to consume certain types of data flowing inside the system and produce one or more analytics outputs based on that data.

[0094] Database 364 may, in some cases, store registered data schemas and incoming data formatted in a pre-registered schema. The schemas may be stored in a table, and a schema ID, for example, a universally unique identifier (UUID) (Type 4), may be used to track the use of that schema by incoming and outgoing data. Data may be stored in separate tables. Various identifiers may be stored, including a schema ID, a generation timestamp, a tenant ID, an OEM ID, a VIN number, and / or other identifiers that may be needed to uniquely identify a data source, in some cases.

[0095] The database 366 may be used to store vehicle information, as described above.

[0096] The database 368 may be used to store the analysis information, as described above.

[0097] A database 370 may be used to store the infrastructure information in the example of FIG.

[0098] Application sync may be used to synchronize data and events. For example, application sync may in some cases use the AWS AppSync protocol, however, other protocols may be used as well.

[0099] 3, application synchronization 372 is used to synchronize between database 364 and data access API 382, ​​among other things, to implement an externally available query API for external parties. For example, the API could be a GraphQL API for external parties that prefer this query language.

[0100] Application Sync 372 may further be used to synchronize between database 366 and vehicle state data access API 384. Vehicle state data access API 384 may be used to query current and past vehicle state. For example, this would allow an external third party to read odometer readings and other useful data. Thus, the system may expose various APIs for accessing either the overall vehicle state or specific vehicle characteristics, depending on the filters defined.

[0101] In some cases, the REST API may query the vehicle database 366 directly, or the data may be retrieved via other APIs, such as the data access API 382.

[0102] 3, application sync 374 is used for synchronization between analytics database 368 and analytics API 380. In particular, various APIs can be used to query analytics generated from the system. This may allow external third parties to read analytics generated from analyzing incoming data from vehicle odometers and other features deemed important for specific use cases. These analytics may be the source for related sample dashboard components such as charts, graphs, and reports, among other options.

[0103] In some cases, a general rest data API 390 may be used to access information in the database 364 .

[0104] Additionally, in some cases, various APIs can be used to access information in a database without using application synchronization. For example, API 388 can access database 370 in the example of FIG.

[0105] Additionally, the authorization / permissions module 392 may be used with the foundation API 388 for various functionality. This functionality may include defining and implementing an API for requesting access to specific data schemas for a tenant. This functionality may further include defining and implementing an API for requesting access to specific attributes common to one or more schemas for a tenant. This functionality may further include defining and implementing an API for granting / denying access requested by a tenant. This functionality may further include refactoring existing data access APIs to check the foundation database 370 to see what access a tenant has and limiting the response to their data access request accordingly. This functionality may further include only allowing requests for schemas that a tenant has permission to access. This functionality may further include filtering out any data to which a tenant does not have access when a tenant has access to only a portion of the data available in a requested data set.

[0106] In some cases, a command line interface 394 may be used to interact with APIs 388 for, for example, configuration, creating or managing permissions, starting, stopping, erasing, or managing VMs (cloud and / or edge VMs), among other functionality.

[0107] However, in other embodiments, a web console may be used to manage the infrastructure rather than, or in addition to, the command line interface 394.

[0108] The APIs in Figure 3 allow third parties and / or OEMs to manage the lifecycle of VMs in the cloud (specifically, creating, starting, stopping, and destroying VMs). This may use authorization mechanisms, which in some cases may include minimal tenant deployment and user tracking mechanisms in a database. In particular, a user wanting to launch a VM may need to upload a Secure Shell (SSH) key, which can be added to the VM when it is created to allow the user to log in to it. This could potentially be done on a tenant basis. This means that these APIs may require a tenant ID, and possibly a session ID, obtained as parameters from a separate login API.

[0109] Service 396 may have an application 398 that interacts with it, which may be used to obtain data from the data service of FIG. 3. For example, service 396 may be an insurance service that allows insurance companies to obtain driving information from users to implement a pay-per-use scheme for insurance. However, other use cases are possible, and this disclosure is not limited to any service or application used with the data service. Service 396 may use API 380, API 382, ​​API 384, and / or API 390 to interact with the data service.

[0110] Using the system of Figure 3, a flexible data pipeline can be defined. A data pipeline can include a network of Lambda functions, and a gateway can determine how to write data to other Lambda functions for processing, streaming, storage, or other functionality. Lambda functions can be created as schemas are added to the system.

[0111] Data can in some cases be cloned between Lambda functions. For example, this can allow access to some data with very low latency by reducing the path for the data to reach its destination while keeping copies in parallel for further processing.

[0112] Data pipelines are scalable; more Lambda functions or generic data pipelines can be replicated as traffic needs require.

[0113] Furthermore, different versions of the data pipeline may exist for different domains, and the functionality of such a data pipeline may vary between different domains.

[0114] Reference is now made to Figure 4, which illustrates a simplified system similar to that of Figure 3. In particular, in the embodiment of Figure 4, the data source may be a vehicle, a virtual machine, or other computing device capable of providing data.

[0115] The source may register its data schema at block 412, for example, through the REST API 338. Specifically, the REST API 338 gateway endpoint and Lambda function may be invoked by the data source 410 at block 412 to register the data schema with the cloud before sending the data. The caller may supply the schema, for example, in the body of a PUT request from the source 410 to block 412, and the response from block 412 to the source 410 may include the schema ID, whether it is a new schema ID obtained from inserting a new schema record into the database of block 412 or the schema ID of an existing database entry for the supplied schema. Block 412 may have a database that includes the schema ID and the corresponding registered schema. In some cases, the schema may be a JSON schema, although other schemas are possible. After registering the schema ID, the source 410 tags the data with the schema ID.

[0116] In the embodiment of FIG. 4, data may be provided to a data injection layer similar to that of FIG. 3. In particular, the data injection layer may include a data injection conduit 334, such as a Kinesis conduit. The data injection layer may further include an IoT core 336 fallback to the data ingestion conduit 334. The data injection layer may further include a REST API 338 as a second fallback. In some cases, the conduit 334 may have knowledge of the data schema and may check whether the data conforms to the schema. If the data does not conform to the schema, the data may be dropped or moved to temporary storage for further processing.

[0117] If data is submitted using the REST API 338, a gateway endpoint and Lambda function (shown as block 414) is provided that acts as a second fallback mechanism for submitting data to the system. Data can be submitted via a PUT request body, which in some cases may be equivalent to what would be sent to Kinesis, or via message attributes of an MQTT message for the submitted IoT core 336 as a first fallback.

[0118] 4, data is fed back from block 414 or IoT core 336 to data ingestion conduit 334. Such data may then be provided to a delivery mechanism 420, such as a Kinesis firehose, in some embodiments.

[0119] The data may then be delivered to data processing pipeline 350. Data processing pipeline 350 first inspects the incoming data packet to ensure that the schema ID is a valid one registered in a database, such as the database of block 412 (if not, the packet should be dropped or moved to temporary storage for further processing). In some cases, data processing pipeline 350 may also check whether the incoming data packet conforms to the schema corresponding to the registered schema ID.

[0120] If the schema ID is valid, the data may be stored in database 364 and then marked as "processed" so that it is transferred to S3 bucket 352. In some cases, data with an invalid schema ID may be delivered anyway, but may be filed in a failure subfolder, as opposed to a different subfolder for successful data.

[0121] Depending on performance requirements, data processing pipeline 350 may need to invoke other processing functions to perform deeper validation of the schema (e.g., whether the data structure actually conforms to the schema) before inserting into the database, and other, more labor-intensive processing may be better performed outside of the initially invoked processing Lambda. In some cases, because validating the schema is labor-intensive, data processing pipeline 350 may sample some data packets to check the schema. Data processing pipeline 350 may query block 412 for registered schema IDs and / or schemas. Data processing pipeline 350 may also be notified by block 412 for any newly registered schema IDs and / or schemas.

[0122] Database 364 is a storage or KeySpace® designed to store incoming data formatted into a pre-registered schema. The incoming data may be stored in tables with various identifiers, including a schema ID, a generation timestamp, a tenant ID, an OEM ID, a VIN number, and / or any other identifier that may be required to uniquely identify the data source.

[0123] Cloud partner API gateway 430 can be used to access data in the system. In particular, such data access can use REST API gateway 430 endpoints and Lambda functions (shown in block 432) that can be invoked by third parties to query data schemas registered with the cloud. Callers can potentially query by name or schema ID, and the response body can include the schema (e.g., a JSON schema). Block 432 can interact with database 364.

[0124] Cloud partner API gateway 430 may also be used to extract data. In particular, data extraction may use REST API gateway 430 endpoints and Lambda functions (shown in block 434) that may be used to extract data from the system (and, in the example of FIG. 4, database 364). Data may be extracted by schema ID, timestamp / data range, and / or possibly other identifiers, for example, if a more limited data set is requested by the caller.

[0125] The embodiments of Figures 3 and 4 can be used in a variety of situations. For example, in the case of insurance, an insurance company may ask a customer for permission to use their data to check their driver rating, for example, to reduce insurance premiums. Instead of using a dongle with an on-board diagnostics (OBD2) port, the system could use synthetic sensors in the vehicle to get the data the insurance company wants.

[0126] The vehicle itself may process the data to create insights, so in some cases the raw data may not need to be passed along, saving on wireless costs. The processed data may go to the cloud before further processing. However, in some cases the raw data may also be sent to the cloud.

[0127] The cloud service may be the insurance company's own cloud or a third-party company's cloud. If it is a third-party cloud, in some cases the data may be transferred to another system, for example using a streaming or Kinesis connection. In this case, the data processing pipeline may cause the data to be stored as raw data or routed for further processing or routed to a different network.

[0128] Depending on the schema, the pipeline may include a low latency pipeline.

[0129] In other situations, the machine learning model may be on-board the vehicle. The vehicle may process data using the machine learning model. However, to update the model, the data may be processed in the cloud for machine learning model retraining. In this case, a pipeline in the cloud may route the data appropriately.

[0130] Other options for use cases are possible.

[0131] Based on the above, a data processing pipeline can be used to route data in a low latency manner by determining a schema and providing the data to further processing lambdas, and possibly cloning the data. Reference is now made to Figure 5.

[0132] 5, data processing pipeline 350 includes gateway function 510. Gateway function 510 is the first point at which the data processing pipeline receives data. Gateway function 510 may check that the schema corresponds to a valid schema ID and may route the data to its destination or further processing lambda based on the schema ID.

[0133] For example, gateway function 510 may, in some cases, send data to routing Lambda 520, which may forward the data to a destination such as an S3 bucket, a streaming service, a connector, among other destinations. In some cases, routing Lambda 520 may clone the received data and provide it to microservice Lambda 540 for further processing.

[0134] Gateway function 510 may further provide the data to S3 Lambda 530 or directly to microservice Lambda 540. The data goes to S3 Lambda 530, so that the data can be stored in its original format in storage, such as storage 352, storage 354, and storage 356. Microservice Lambda 540 may process the original cloned data. The processed data can be stored in storage, such as storage 352, storage 354, storage 356, and storage 364, or sent to message broker 360 for further processing.

[0135] In some cases, Lambdas can be cascading, and depending on the data schema, data can be routed or cloned as required.

[0136] Either the gateway function 510 or the processing functions may be replicated or parallelized to adapt to traffic loads and processing demands. For example, when the amount of data reaches a threshold, parallel data pipelines may be created, thereby allowing the system to scale with increased data loads. Additionally, in some cases, when the amount of data reaches a threshold, any second processing function (e.g., microservice Lambda 540, etc.) may be replicated in the data pipeline, allowing the system to scale with increased data loads.

[0137] Processing in gateway function 510 allows high latency functions, such as database storage (which may intermittently introduce latency) or S3 storage, which may be high latency operations, to be separated from low latency operations, thereby creating a faster pipeline.

[0138] Additionally, in some cases, data with a particular schema identifier may be routed by the gateway function 510 to a bus that may act as a real-time data cache that may be queried.

[0139] In some cases, the schema ID can be used to subscribe data to different topics, in some cases one topic per schema. Thus, gateway function 510 can route data based on schema, and thus topic, and gateway function 510 can be triggered to process incoming data as it arrives in a topic of interest. This event-driven logic can result in massive parallelization.

[0140] Additionally, any generated data can be fed back onto the bus for further downstream processing by components using the data processing pipeline 350, while simultaneously being branched off into slower data lines for upstream archiving, storage, and external streaming, similar to how data was previously branched off in flow from the general data pipeline.

[0141] Using such a gateway, the process is illustrated with respect to Figure 6. In particular, the process begins at block 610 and proceeds to block 620 where the system receives a new schema for data (i.e., the schema has not been previously registered).

[0142] Based on the schema, the process may proceed to block 630 to create a new processing function, such as a Processing Lambda, for that schema. The Processing Lambda may be added to the data processing pipeline, and data in the pipeline with the new schema ID may be routed through the new processing function, either directly from the gateway function or through another processing function, as shown in block 640. In some cases, the data may be a clone of the original data, and the original data may be routed using different processing functionality.

[0143] The process proceeds from block 640 to block 650 and ends.

[0144] Hardware

[0145] The above nodes, regions, computing devices, platforms, electronic control units, or other computing systems may be implemented using any computing device. A simplified diagram of one of the computing devices is shown with reference to FIG. 7. The computing device of FIG. 7 may be any fixed or mobile computing device. Although the embodiment of FIG. 7 shows only a single computing device, multiple computing devices may be used with the systems and methods herein, and the embodiment of FIG. 7 is provided merely as an example.

[0146] 7, device 710 includes a processor 720 and a communication subsystem 730, which cooperate to implement the methods of the above embodiments. The communication subsystem 730 enables device 710 to communicate with other devices or network elements and may vary based on the type of communication being implemented. Furthermore, communication subsystem 730 may include multiple communication technologies, including any wired or wireless communication technology.

[0147] Processor 720 is configured to execute programmable logic (and shown in the example of FIG. 7 as memory 732), which may be stored along with data on device 710. Memory 732 may be any tangible, non-transitory computer-readable storage medium that stores instruction code that, when executed by processor 720, causes device 710 to perform the methods of the present disclosure. The computer-readable storage medium may be a tangible or transitory / non-transitory medium, such as, for example, optical (e.g., CD, DVD, etc.), magnetic (e.g., tape), flash drive, hard drive, or other memory known in the art.

[0148] Instead of, or in addition to, memory 732, device 710 may access data or programmable logic from an external storage medium, for example, via communications subsystem 730.

[0149] 7, one or more sensors 740 may be associated with the computing device, however, this is optional and in some cases the computing device 710 is not associated with any sensors.

[0150] Communication between the various elements of device 710 may, in one embodiment, be through an internal bus 760. However, other forms of communication are possible.

[0151] The embodiments described herein are examples of structures, systems, or methods having elements that correspond to elements of the technology of the present application. This written description may enable one skilled in the art to make and use embodiments having alternative elements that similarly correspond to elements of the technology of the present application. Thus, the intended scope of the technology of the present application includes other structures, systems, or methods that do not differ from the technology of the present application described herein, and further includes other structures, systems, or methods with insubstantial differences from the technology of the present application described herein.

[0152] Although operations are depicted in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown, or that all of the illustrated operations be performed, to achieve desired results. In some situations, multitasking and parallel processing may be employed. Furthermore, the separation of various system components in the above implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0153] Additionally, techniques, systems, subsystems, and methods described and illustrated in various implementations as separate or distinct may be combined or integrated with other systems, modules, techniques, or methods. Other terms shown or described as coupled or directly coupled or in communication with each other may also be indirectly coupled or in communication, electrically, mechanically, or otherwise, through some interface, device, or intermediate component. Other examples of modifications, substitutions, and alterations will be apparent to, and may be made by, those skilled in the art.

[0154] While the foregoing detailed description illustrates, describes, and points out the basic novel features of the present disclosure as applied to various implementations, it will be understood that various omissions, substitutions, and changes in the form and details of the illustrated systems may be made by those skilled in the art. Additionally, no order of method steps is implied by the order presented in the claims.

[0155] When messages are sent to / from an electronic device, such actions need not be immediate or directly from a server. They may be delivered synchronously or asynchronously from a server or other computing system infrastructure supporting the devices / methods / systems described herein. The above steps may involve, in whole or in part, synchronous or asynchronous communication to / from the device / infrastructure. Furthermore, communication from the electronic device may be to one or more endpoints on a network. These endpoints may be serviced by a server, a distributed computing system, a stream processor, or the like. A content delivery network (CDN) may also provide communication to the electronic device. For example, rather than a typical server response, the server may set up or direct data for the content delivery network (CDN) to await subsequent download by the electronic device, e.g., upon subsequent activity of the electronic device. Thus, data may be sent directly from the server or from another infrastructure, e.g., a distributed infrastructure or CDN, as part of or separate from the system.

[0156] Typically, the storage medium may include any or some combination of semiconductor memory devices, such as dynamic or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; magnetic disks, such as fixed disks, floppy disks, and removable disks; other magnetic media, including tape; optical media, such as compact disks (CDs) or digital video disks (DVDs); or other types of storage devices. The instructions described above may be provided on one computer-readable or machine-readable storage medium, or alternatively, may be provided on multiple computer-readable or machine-readable storage media, possibly distributed across a larger system having multiple nodes. Such computer-readable or machine-readable storage medium(s) is / are considered to be part of an article (or manufacture). An article or manufacture may refer to any manufactured single component or multiple components. The storage medium(s) may be located either within the machine that executes the machine-readable instructions, or may be located at a remote site where the machine-readable instructions can be downloaded over a network for execution.

[0157] In the preceding description, numerous details are set forth to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations from the above-described details. It is intended that the appended claims cover such modifications and variations.

Claims

1. 1. A method in a data pipeline in a computing system, the method comprising: receiving data and a schema identifier associated with the data; verifying that the schema identifier is valid; and routing the data in the data pipeline based on the schema identifier; A method comprising:

2. The method of claim 1 , wherein the routing comprises routing the data to a second processing function in the data pipeline.

3. The routing step comprises: cloning the data to create cloned data; transmitting the data to a destination; transmitting the clone data to a processing function; 2. The method of claim 1, wherein the processing function includes a slower path than transmitting the data to the destination.

4. receiving a new schema identifier; determining a data path for data associated with the new schema identifier; The method of claim 1 further comprising:

5. The method of claim 4 , further comprising creating a new processing function for data associated with the new schema identifier.

6. The method of claim 1 , wherein a parallel data pipeline is created when an amount of data reaches a threshold.

7. The method of claim 2 , wherein the second processing function is replicated in the data pipeline when an amount of data reaches a threshold.

8. The method of claim 1 , wherein verifying that the schema identifier is valid comprises checking that a schema of the data matches a schema of the schema identifier.

9. 10. The method of claim 1, wherein the routing of the data comprises routing the data to a bus that includes a real-time data cache.

10. a processor; A data pipeline and Communication subsystem and 1. A computing system comprising: receiving data and a schema identifier associated with the data; using said processor to verify that said schema identifier is valid; routing the data in the data pipeline based on the schema identifier; 1. A computing system configured to:

11. The computing system of claim 10 , wherein the computing system is configured to route by routing the data to a second processing function in the data pipeline.

12. The computing system includes: cloning the data to create cloned data; transmitting the data to a destination; transmitting the clone data to a processing function; 11. The computing system of claim 10, wherein the processing function includes a slower path than transmitting the data to the destination.

13. The computing system includes: receiving a new schema identifier; determining a data path for data associated with the new schema identifier; The computing system of claim 10 further configured to:

14. The computing system of claim 13 , wherein the computing system is further configured to create a new processing function for data associated with the new schema identifier.

15. The computing system of claim 10 , wherein a parallel data pipeline is created when an amount of data reaches a threshold.

16. 12. The computing system of claim 11, wherein the second processing function is replicated in the data pipeline when an amount of data reaches a threshold.

17. 11. The computing system of claim 10, wherein the computing system is configured to verify that the schema identifier is valid by checking that a schema of the data matches a schema of the schema identifier.

18. The computing system of claim 10 , wherein the computing system is configured to route the data by routing the data to a bus that includes a real-time data cache.

19. 1. A computer-readable medium for storing instruction code, the instruction code, when executed by a processor in a computing system, receiving data and a schema identifier associated with the data; using said processor to verify that said schema identifier is valid; routing the data in the data pipeline based on the schema identifier; a computer-readable medium for causing the computing system to perform the steps of:

Images