Method and system for vehicle data file playback

Abstract

1. A method in a computing device, the method including: receiving sensor data from a data source at the computing device; converting the sensor data into transformed data at a playback tool on the computing device, thereby bypassing an abstraction layer at the computing device; and providing the transformed data to at least one synthetic sensor on the computing device, each of the at least one synthetic sensor providing insight into operation of the computing device.

Description

Technical Field

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Application Serial No. 17 / 480,801, filed on September 21, 2021. The disclosure of the prior application is considered part of the disclosure of the present application (and is incorporated herein by reference).

[0002] (FIELD OF THE DISCLOSURE) The present disclosure relates to distributed systems, and in particular to data file playback in distributed systems.

Background Art

[0003] Modern vehicles include many sensors. However, such sensors may be distributed across various computing nodes on the vehicle, and each computing node may have access to zero, one, or more than one sensor driver. Such sensor nodes may further belong to different manufacturers and operate using different operating systems. Similarly, other distributed systems may include a plurality of nodes that require communication with each other.

[0004] A sensor or a group of sensors can be used to generate information that may be useful for one or more applications. Such information is referred to herein as insights.

[0005] However, for prototyping and safety reasons, development may need to be performed separately from an actual vehicle.

Summary of the Invention

Means for Solving the Problems

[0006] The present disclosure will be more thoroughly understood with reference to the accompanying drawings.

Brief Description of the Drawings

[0007] [Figure 1]Figure 1 is a block diagram showing an exemplary computing node in a computer system.

[0008] [Figure 2] Figure 2 is a block diagram showing the first simplified playback architecture.

[0009] [Figure 3] Figure 3 is a block diagram showing the second simplified playback architecture.

[0010] [Figure 4] Figure 4 is a block diagram showing a third simplified playback architecture.

[0011] [Figure 5] Figure 5 is a process diagram illustrating the use of a playback tool in a computing device.

[0012] [Figure 6] Figure 6 is a simplified block diagram of a computing device that can be used in conjunction with embodiments of the present disclosure. [Modes for carrying out the invention]

[0013] This disclosure provides a method in a computing device, the method comprising: receiving sensor data from a data source in the computing device; converting the sensor data into converted data in a playback tool on the computing device, thereby bypassing the abstraction layer in the computing device; and providing the converted data to at least one synthetic sensor on the computing device, each of which provides insights into the operation of the computing device.

[0014] This disclosure further provides a computing device comprising a processor and a communication subsystem, wherein the computing device is configured to receive sensor data from a data source, convert the sensor data into transformed data in a playback tool on the computing device, thereby bypassing the abstraction layer in the computing device, and providing the transformed data to at least one synthetic sensor on the computing device, each of which provides insights into the operation of the computing device.

[0015] The disclosure further provides a computer-readable medium for storing instruction code, which, when executed by the processor of the computing device, causes the computing device to receive sensor data from a data source, convert the sensor data into transformed data in a playback tool on the computing device, thereby bypassing the abstraction layer on the computing device, and provide the transformed data to at least one synthetic sensor on the computing device, each of which provides insights into the operation of the computing device.

[0016] In modern vehicles, information from one or more physical sensors can be processed to generate "insights" that may be useful to the system. Such one or more physical sensors and the processing associated with them can logically be referred to as microservices or synthetic sensors (SS). The terms microservices and synthetic sensors are used synonymously herein.

[0017] Synthetic sensors may exist in other types of applications, including, but not limited to, medical applications, manufacturing applications, and Internet of Things applications, and this disclosure is not limited to automotive applications. Automotive applications are provided below for illustrative purposes.

[0018] Insight is a term used herein to describe any computer-generated interpretation of basic sensor data. Insights can be as simple as data sets or correlations, or as complex as artificial intelligence and machine learning. For example, a temperature sensor that provides high and low thresholds for notification may be considered an "insight." With respect to location services, geofencing is an insight. With respect to cameras, occupant recognition can be an insight. The use of a combination of sensors, such as a temperature sensor and a camera, can be used with an artificial intelligence model to determine whether a car seat is occupied in a hot vehicle, and this can be an insight. Many other examples of insights are possible.

[0019] In one embodiment, a vehicle application may be implemented in a system that provides consistent access to vehicle data and intelligent insights in a manner familiar and accessible to the developer community. Such an environment could enable cloud developers to extend their reach to the edge within the vehicle by developing synthetic sensors that derive intelligent insights about vehicle data, using common cloud development techniques and paradigms. Such an environment can provide consistent access to vehicle data, thereby allowing synthetic sensors to be written to and deployed across a wide vehicle base without the need for bespoke customization.

[0020] Insights may arise based on processors running on a first facility or domain, which often need to be shared with authorized software modules running within an external domain. The first domain controls the installation and communication of modules within it, and therefore can determine their identification and authorization. However, the first domain does not have such control over other domains.

[0021] For example, one external domain may be an automotive in-vehicle infotainment (IVI) system that can execute an application that accesses insights and provides information to a driver.

[0022] However, prior to deploying such systems in the real world, it may be desirable to playback data files within the system, for example, as a simulation. This may be due to various factors including safety reasons, rapid prototyping, or demonstration of the system to potential customers, developers, partners, or other parties.

[0023] For example, deploying a new synthetic sensor into a real-world environment requires development of an abstraction layer to integrate the sensor into the system, requires data to be obtained from the system to ensure that the system is operating correctly, and may potentially cause errors or unsafe conditions in the vehicle.

[0024] Furthermore, this can be difficult to achieve in an actual vehicle in order to demonstrate the functionality of a synthetic sensor or multiple synthetic sensors to intended users.

[0025] (Exemplary Vehicle System)

[0026] The present disclosure will be described with reference to an automotive system including nodes. However, this is provided merely for illustrative purposes, and the methods and systems described herein may equally be used with any other system.

[0027] For example, reference is now made to FIG. 1, which shows node 110. As used herein, a node may be one or a group of an electronic control unit, a central processing unit, or a kernel control, among other options, and may be considered a single computing unit.

[0028] In the example in Figure 1, node 110 includes a service manager 120 that can interact with drivers for the sensors to which the node is connected. For example, node 110 may have access to location sensors such as a Global Positioning System (GPS) chipset, as shown in block 122.

[0029] A hardware abstraction layer (HAL) may be provided on node 110 to enable node 110 to interact with modules on other nodes and to functionally equip the computing system, comprising HAL services 130. Each HAL service 130 is involved in sensor integration and provides various functions, including integration with lower-level sensors and normalization of sensor data; and / or, if required, can provide a barrier between safety-certified software and non-certified software. Other functions relating to HAL services are also possible.

[0030] In the example in Figure 1, the HAL is provided for camera information, as shown in block 132.

[0031] The example in Figure 1 shows node 110 with a single service and a single HAL, but it is provided solely for illustrative purposes. Node 110 may have a single service without a HAL, a single HAL without a service, multiple services without a HAL, multiple HALs without services, multiple HALs without services, and / or combinations of services and HALs.

[0032] One example of a system that could utilize Node 110 would be an application development environment for vehicles. Such an application development environment could develop applications for user experience, including comfort, navigation, and infotainment; applications for safety, fleet management, performance monitoring, or other such applications for the vehicle environment. In particular, vehicles provide multiple sensors, and different manufacturers, models, or brands use different sensors with different data formats or values, which can generate fragmented sensor measurements depending on the sensors. This fragmentation hinders the advancement of application ecosystems that utilize vehicle data. In addition, low-level sensor data is often too fine-grained to be easily useful for applications.

[0033] In this regard, a hardware abstraction layer can be used to provide a hardware-independent sensor interface / abstraction that can encapsulate interactions with the underlying sensor driver. The use of hardware abstractions across various computing nodes can create an extensible platform, provide barriers between modules, and, for example, enhance the isolation of a security authentication system from other systems, among other options.

[0034] Applications do not directly interact with the sensor hardware to access sensor data; instead, they leverage a hardware abstraction layer. This separation provides a clear distinction between the involvement of the HAL (Hardware Integration and Normalization of Sensor Data) and other abstractions such as the Vehicle Abstraction Layer (VAL), which is used to manage access to vehicle data and provide value-added insights.

[0035] Specifically, insights can leverage sensor data from multiple HALs to provide vehicle abstractions and value-added insights. Vehicle insight services in VAL can control access to vehicle data in a normalized form and provide value-added inferences. Examples of insight services may include, among others, a location service that can provide coordinate location data and insights such as geofencing in a consistent format; a seat service that can provide a multitude of seat information such as belt status, weight, location, and child lock status; a camera service that can provide video streams for in-vehicle cameras and potentially functions such as transformation and / or cropping; a battery service that can provide insights and access to the battery, such as charge status / consumption / expected remaining time / expected mileage; and a door service that can provide abstractions regarding vehicle doors and door status.

[0036] Insight services can leverage sensor data from multiple HALs to provide vehicle abstraction and value-added insights. Higher levels of insight into the data enable application developers to generate future automotive experiences. Insight is a term used to describe any value-added interpretation of basic sensor data. Insight can be as simple as a data set or correlation, or as complex as artificial intelligence and machine learning. For example, with respect to a temperature sensor, providing high and low water level indicators for notification could be considered an "insight." With respect to location services, geofencing is an insight, and with respect to cameras, occupant recognition could be considered an insight.

[0037] A node with a service and HAL manager, as illustrated in Figure 1, is therefore required for such an application development environment. In many systems, node 110 is required to communicate with other nodes, such as other ECUs, CPUs, or computing systems, which may use a different operating system than that of node 110.

[0038] However, when developing, testing, or demonstrating synthetic sensors, the overhead of generating an abstraction layer between real-world sensor data and the synthetic sensor can be burdensome and hinder development. In this regard, a playback mechanism can be used to bypass the hardware and vehicle abstraction layer, and such a playback mechanism can be used for prototyping or demonstrating synthetic sensors.

[0039] (Proof-of-concept and prototyping tools)

[0040] According to embodiments of this disclosure, playback of data relating to vehicle operation, particularly data relating to synthetic sensors within the vehicle, can be performed using a data source that provides information to various abstraction layer playback tools within edge nodes. This is illustrated, for example, with reference to Figures 2, 3, and 4.

[0041] A reference is now made to Figure 2, which shows a simplified architectural diagram relating to the first playback architecture. In the example of Figure 2, the data source 210 may be any computer system or database in some embodiments. For example, the data source may include, among other things, sample video clips stored in the database, data on specific sensors such as sample temperature measurements relating to a simulation of the inside of a vehicle, and a correlation between a sample route traveled and the battery level in an electric vehicle.

[0042] In other embodiments, the data source 210 may be a real sensor that provides feedback to a computer node running the synthetic sensor, for example, through a communication subsystem. For example, the data source 210 may include a camera connected to a real vehicle that otherwise provides feedback to a playback system, and sensors such as a temperature sensor may be connected to a computer running the edge node, for example, through a USB or short-range communication port. Other options for the data source 210 are also possible.

[0043] In the example in Figure 2, data source 210 contains Vehicle Energy Dataset (VED) data 212, which is the name given to data collected in 2016 / 2017 from 383 private vehicles in Ann Arbor, Michigan. The VED data 212 is provided to edge node 220, which runs a simulation of the synthetic sensor. In particular, edge node 220 contains a Service Software Development Kit (SDK) 230, which includes various hardware abstraction layers and a vehicle abstraction layer.

[0044] In the example in Figure 2, the Global Positioning System (GPS) HAL service 232 may be provided within the service SDK 230. As provided above, the HAL service is involved in sensor integration and provides various functions including integration into underlying data, normalization of sensor data; and / or, if required, it may provide a barrier between safety-certified software and non-certified software. Other functions relating to the HAL service are also possible. This disclosure is not limited to the use of any particular HAL service and GPS HAL service and is provided for illustrative purposes only.

[0045] Therefore, in the embodiment shown in Figure 2, the GPS HAL service 232 can convert the VED data into a data type, format, rate, etc., suitable for a specific composite sensor. Such data is then provided to the localization VAL service 234, which is used to manage access to vehicle data and provide value-added insights.

[0046] Specifically, as provided above, insights can leverage sensor data from multiple HALs to provide vehicle abstractions and value-added insights. The vehicle insights service within VAL can control access to normalized form vehicle data and provide value-added inference.

[0047] However, in other embodiments, the hardware abstraction layer may not be written for a particular sensor or sensor type. As shown above, this can lead to significant development delays in writing the abstraction layer in a specific format, based on coding rules and requirements for the distributed system.

[0048] In this regard, if an abstraction layer is not present, the playback tool 236 may be used to bypass the abstraction layer. The playback tool used herein may be a lightweight and extensible module that can be programmed for specific data sources (sensors) and synthetic sensors that utilize the data. Thus, the playback tool provides a mechanism for easily plugging in new and synthetic sensors and prototyping or demonstrating the environment.

[0049] In the example in Figure 2, the playback tool 236 includes a sensor module 238, which is illustrated to utilize the OpenXC format. However, the OpenXC format is provided for illustrative purposes only, and any formatting of the data source may be used for playback. The sensor module 238 is designed to take input from a specific data source and provide output in a specific format designed for the synthetic sensor. For example, this may include, among other factors, the sampling rate, data format type, and normalization of the sensor data. Since the data type, formatting, mileage, and other information from the data file or known sensor are known to the developer of the playback module 238, and the data formatting, rate, normalization, and other information required by the synthetic sensor are also known to the developer, the module 238 can be easily generated.

[0050] Furthermore, module 238 may have variables, transformation algorithms, filters, sampling rates, or other parameters that can be set at runtime. This may be done for performance optimization of the synthetic sensor 240 for rapid testing of the synthetic sensor. As will be understood by those skilled in the art, such parameters may ultimately be used to generate hardware and vehicle abstraction layers for the system with respect to a particular sensor type, and by having such capability for playback, this may improve and accelerate the task of generating the abstraction layers.

[0051] In the embodiment shown in Figure 2, the composite sensor 240 is a driver behavior composite sensor. This is provided solely for illustrative purposes, and other types of composite sensors may also be used in conjunction with embodiments of the present disclosure.

[0052] A location service client 242 is located within the composite sensor harness 241. The location service client 242 can query location data and, if available, retrieve it from the location VAL service 234. This location data can then be provided to the driver behavior module 244.

[0053] Alternatively, if the playback module 238 is used, the data can be directly inserted into the module within the synthetic sensor 240 that requires such data, thereby bypassing the hardware abstraction requirement.

[0054] The driver behavior module 244 can use the machine learning module 246 to derive insights about driver behavior and provide such insights to be returned to the driver behavior module 244. Based on the driver behavior module 244 compiling such data, the driver behavior proxy 248 may receive the data.

[0055] The driver behavior proxy 248 may provide such data to the insight consumer 250. For example, the insight consumer 250 may be a dashboard driver behavior display 252 that can provide a visual representation of driver behavior. In other cases, the insight consumer 250 may be an insurance company that has a contract with the driver to obtain driver behavior information. Other options regarding the insight user 250 are also possible.

[0056] In one simulation, the insight consumer 250 could be another program connected to the edge node 220 to provide feedback to the user or target audience, for example, through a visual display, data stream, and other options.

[0057] In some embodiments, control over the start and stop of various synthetic sensors may be desirable. In this case, trigger services and lifecycle services may be provided within the computing device. A reference to Figure 3 is made here.

[0058] In the embodiment shown in Figure 3, the data source 310 may be a database or computer file containing information about one or more sensors that can be provided to the computing device 320. For example, in the embodiment shown in Figure 3, the data script file 312 may include a localization data OpenXC file 314. As will be understood by those skilled in the art, an OpenXC file is an application programming interface (API) for automobiles, an open standard proposed for communication between various hardware elements. Thus, the simplified example in Figure 3 provides a localization-based behavior service, and the localization data is provided through the openXC file 314.

[0059] Data 314 from the OpenXC file can be provided to the service SDK 324 on the computing device 320, more specifically to the GPS HAL service 325. The service SDK 324 resides within a first operating system 322 simulated on the computing device 320. For example, in some embodiments, the first operating system may be the BlackBerry® QNX operating system. However, other operating systems may be used equally.

[0060] A Global Positioning System (GPS) HAL service 325 may be provided within the service SDK 324. As provided above, the HAL service is involved in sensor integration and provides various functions including integration with underlying sensors, normalization of sensor data; and / or may provide a barrier between safety-certified software and non-certified software, if required. Other functions relating to the HAL service are also possible. This disclosure is not limited to the use of any particular HAL service and GPS HAL service and is provided for illustrative purposes only.

[0061] Therefore, in the embodiment shown in Figure 3, the GPS HAL service 325 converts the location data OpenXC information into a data type, format, rate, etc., suitable for a specific synthetic sensor. Such data is then provided to the location VAL service 326, which is used to manage access to vehicle data and provide value-added insights.

[0062] If an abstraction layer is not present, the playback tool 328 may be used to bypass the abstraction layer. Similar to the playback tool 238 in the embodiment of Figure 2, the playback tool 328 may be a lightweight and extensible module that can be programmed for specific data sources (sensors) and synthetic sensors that utilize the data. In this way, the playback tool provides a mechanism for easily plugging in new and synthetic sensors and prototyping or demonstrating environments.

[0063] In the example in Figure 3, the playback tool 328 includes a sensor module 329, which is illustrated to utilize the OpenXC format. However, the OpenXC format is provided for illustrative purposes only, and any formatting of the data source may be used for playback. The sensor module 329 is designed to take input from a specific data source and provide output in a specific format designed for the synthetic sensor. For example, this may include, among other factors, the sampling rate, data format type, and normalization of the sensor data. Since the data type, formatting, mileage, and other information from the data file or known sensor are known to the developer of the playback tool 328, and the data formatting, rate, normalization, and other information required by the synthetic sensor are also known to the developer, the playback tool 328 can be easily generated.

[0064] Furthermore, the playback tool 328 may have variables, transformation algorithms, filters, sampling rates, or other parameters that can be set at runtime. This may be done for performance optimization of the synthetic sensor 340 for rapid testing of the synthetic sensor. As will be understood by those skilled in the art, such parameters may ultimately be used to generate hardware abstraction layers and vehicle abstraction layers for the system with respect to a particular sensor type, and by having such capability for playback, this may improve and accelerate the task of generating the abstraction layers.

[0065] The localization VAL service 326 may trigger a trigger service 332 in the synthetic sensor and insight library 330 when it is provided with new information from the abstraction layer in the SDK service 324. Such a trigger service may then cause the lifecycle service 334 to start the runtime on the behavioral characteristics synthetic sensor 340. In particular, the synthetic sensor 340 may have a runtime library 344 which includes a library 342 and can be started by the lifecycle service 334.

[0066] The runtime 344 includes location information received from the service SDK and may provide such location information to a location in task processing or block 346. The location task block 346 may communicate with the machine learning block 348, which may also retrieve data from the static data store 350 to determine behavioral characteristics.

[0067] The machine learning module 348 can further communicate with a machine learning service client 352 for controlling the machine learning module 348.

[0068] If a hardware abstraction layer does not exist for a particular sensor, a playback tool 328 may be used. In this case, the output from the sensor module 329 can be directly inserted into the synthetic sensor 340. This is shown, for example, as data directly input into the localization task 346. This avoids architectures that provide a trigger when new data is available, start the lifecycle, and run the synthetic sensor. In practice, the synthetic sensor 340 may be set to run continuously for demonstration or prototyping purposes, thereby eliminating the need for trigger and lifecycle services for sensors that do not yet have an abstraction layer written for them.

[0069] The localization task 346 may further feed insights to the insight broker 360. The insights are fed into the simulator as a closed loop from the test viewpoint to have data adjustments. In particular, the insight broker 360 may provide a loop back to the trigger service 332 to control the trigger of the synthetic sensor 340. The insight broker 360 may further communicate with the insight consumer as described below.

[0070] The machine learning manager 362 is part of the synthetic sensor and insight library 330 and can control the machine learning service client 352.

[0071] In the example shown in Figure 3, a second operating system 370 may reside on the computing device 320. For example, the computing device 320 may simulate the Android® operating system to replicate an Android® in-vehicle infotainment (IVI) system. However, other operating systems and applications can also be simulated.

[0072] The operating system 370 includes an application development library 372 which includes a bridge 374 that can be used to communicate between the first operating system and the second operating system. In particular, the insight broker 360 can communicate with the bridge 374.

[0073] Once data is received through bridge 374 on operating system 370, the data can be provided to application 376. As shown above, an example of an application may be an IVI application on a vehicle. This includes a user interface 378 that can be used to display information based on insights obtained from synthetic sensors within the first operating system.

[0074] Therefore, the embodiment shown in Figure 3 provides a simple, open, adaptable, and extensible simulator that can use modules commonly found on a vehicle to control data simulation and utilize trigger services, lifecycle services, and insight broker services to simulate the operation of those modules. The data can be in any format including open APIs such as the OpenXC format, which can then be converted into data suitable for the synthetic sensor through one or more abstraction layers.

[0075] A more detailed architecture is shown with respect to Figure 4. The architecture in Figure 4 is provided with an example of an electric vehicle (EV) simulation that provides mileage expert output to the user interface. However, this is merely an example, and other architectural options are also possible.

[0076] In the example in Figure 4, the data source 410 may include multiple data script files 412. These may include an initial data file 414, which may include various criteria such as destination location, USB status or Wi-Fi status, screen status, and HVAC status, among other options. In one embodiment, the initial data 414 may utilize the OpenXC protocol.

[0077] Other data that may be present in the data script file 412 may include electric vehicle data 416. For example, this may include the battery status of the vehicle as it travels along a simulated route.

[0078] Further data may include video data 418. Such video data may be used for tasks such as driver identification verification, as described below.

[0079] Further data within the data source 410 may include localization data 420, which in some embodiments may be in the OpenXC file format. However, other file formats are also possible.

[0080] In the embodiment shown in Figure 4, the data source may be part of the computing device 430 or may be separate from it. The computing device 430 may have a first operating system 432, which may simulate such a first operating system. For example, the first operating system may be the BlackBerry® QNX operating system in some embodiments. However, other options regarding the first operating system 432 are also possible.

[0081] Within the first operating system 432, a playback tool 433 may exist. The playback tool 433 may be used to transform data from sensors that do not have an abstraction layer written for such sensors. As will be understood by those skilled in the art, writing an abstraction layer including HAL and VAL is time-consuming, and therefore the playback tool 433 may be used to bypass the generation of such an abstraction layer for prototyping purposes. Specifically, the playback tool 433 provides a lightweight and adaptable tool to enable different sensors (whether real or virtual) to be plugged into the computing device 430 without the need to write such an abstraction layer, and thus increases the speed of prototyping and testing synthetic sensors or providing demonstrations to users who will utilize such synthetic sensors.

[0082] In the embodiment shown in Figure 4, the playback tool 433 has an EV OpenXC module 434 to receive EV data from the EV data source 416.

[0083] The simulator 433 may further include, for example, a converter 436 for converting video data 418. The converter 436 can therefore convert one file format to a second file format, such as MP4 to JPEG, among other options. In some embodiments, rather than converting to a specific data format, the converter 436 may convert the file into a mathematical matrix, for example, for direct input to a machine learning or artificial intelligence module in a synthetic sensor.

[0084] The output from the converter can be provided to the camera OpenCV module 438, which can use the OpenCV protocol to provide the output.

[0085] Furthermore, in some embodiments, a service SDK 440 with various abstraction layers may also be present within the computing device 430. In the embodiment shown in Figure 4, the service SDK 440 includes a GPS hardware abstraction layer service 442 and a localization vehicle abstraction layer service 444.

[0086] The embodiment in Figure 4 further includes a synthetic sensor and insight library 450, which may include a trigger service 452 and a lifecycle service 454. The trigger service 452 may be used to trigger the start or stop of the synthetic sensor. It may be further used as a trigger for starting the vehicle, among other options, or it may be triggered based on input from the insight broker. The lifecycle service 454 may be used to start and stop the sensor throughout the playback lifecycle.

[0087] In the embodiment shown in Figure 4, three composite sensors are provided. Specifically, a behavior change composite sensor 460 is provided, which can be used, for example, to provide a behavior change recommendation when an electric vehicle is within proximity of a threshold distance required to complete a run.

[0088] Furthermore, a driver identification synthesis sensor 470 is provided, which can be used to identify the driver and thus configure the vehicle based on the driver profile.

[0089] The feature modification synthesis sensor 480 can be used to recommend various feature modifications to the driver, which may, for example, extend battery life when the predicted mileage compared to route requirements falls below a threshold.

[0090] The behavior-changing synthetic sensor 460 may have its own sensor and an insight library 462, the insight library 462 may include a synthetic sensor runtime module 464 and a machine learning service client 465.

[0091] Furthermore, the behavior change synthetic sensor 460 may have multiple blocks or processes that can be used to generate insights. For example, the behavior change synthetic sensor 460 may have an EV task block 466 that can monitor EV tasks. The synthetic sensor may also have a location task 467 that can monitor the vehicle's location relative to a destination. The behavior change recommendation engine machine learning module 468 may compare EV tasks from the EV task block 466 with location tasks from the location task block 467 and provide behavior recommendations regarding the vehicle's driving based on the expected mileage and the distance to the destination.

[0092] The driver identification synthetic sensor 470 includes a behavior and insights library 472 which may include a synthetic sensor runtime 474 and a machine learning service client 475.

[0093] In this case, the synthetic sensor may include a camera task 476 to process camera input. Furthermore, the driver identification machine learning module 477 may use the input from the camera task and identify the driver based on input from the driver profile database 478. The driver profile database 478 may, in some cases, include driver profile preferences for multiple drivers.

[0094] The feature modification synthesis sensor 480 may be used to recommend feature modifications. In this case, the feature modification synthesis sensor 480 includes a sensor and an insight library 482 having a synthesis sensor runtime module 484. Furthermore, the library may include a machine learning service client 485.

[0095] The synthetic sensor may have a location task module 486 to monitor the vehicle's location. Furthermore, the sensor may have an EV task module 487 to monitor the vehicle's battery.

[0096] Furthermore, task module 489 may be used to provide a list of tasks being used by the vehicle.

[0097] The feature modification recommendation engine machine learning module 488 can receive input from various task modules and provide feature modification recommendations.

[0098] The machine learning manager 490 can manage various machine learning service clients 465, 475, and 485.

[0099] The insight broker 492 can provide a closed loop for data adjustment and can be used for test perspectives.

[0100] In the embodiment shown in Figure 4, a second operating system may be used. For example, the second operating system may be the Android® operating system and an in-vehicle infotainment system. However, other options for both the operating system and the applications are also possible. In particular, the operating system 494 includes an application development library 495 with a bridge 496 to provide interaction between the first operating system and the second operating system.

[0101] An electric vehicle mileage application 497 is shown as an exemplary application and may include a user interface component 498 for providing an output to the vehicle user, whether visual, tactile, or auditory.

[0102] During operation, initial data 414 can be provided directly to the synthetic sensor to set the state within the synthetic sensor. Thus, location data, destination data, HVAC state, screen state, and Wi-Fi state can be provided to the feature change recommendation engine machine learning module 488 and the behavior change recommendation engine 468.

[0103] Next, playback can be initiated, and EV data can be provided from the EV data source 416 to the EV OpenXC simulator module 434. Subsequently, EV tasks 466 and 487 can query the EV OpenXC module 434 to find the EV state. As will be understood, this bypasses the trigger service 452 and the lifecycle service 454, thus enabling rapid development, prototyping, and demonstration.

[0104] The video data 418 can be provided to the camera OpenCV module 438 through the converter 434. The camera task module 476 can then interact with the simulator module and acquire the camera data. Again, this bypasses the trigger service 452 and the lifecycle service 454.

[0105] Similarly, location data from location data source 420 may be provided through GPS HAL service 442 and location VAL service 444, and the output from location VAL service may be provided to trigger service 452. This can trigger the trigger service 452 to start lifecycle service 454 when the vehicle is moving.

[0106] Lifecycle service 454 may provide inputs to runtime modules 464, 474, and 484 to indicate to the synthetic sensor that the synthetic sensor should start operating.

[0107] As described above, within each synthetic sensor, various task modules may be used by machine learning modules to provide insights. For example, in a driving scenario, the behavior change synthetic sensor 460 may provide insights from the recommendation engine machine learning module 468 to indicate that if the electric vehicle does not have enough mileage to reach its destination, the driver behavior should be changed, or the vehicle should be recharged prior to reaching its destination. Similarly, the driver identifier machine learning module 477 may capture an image of the vehicle's driver, then process the image and utilize the camera data to identify the driver, and thus utilize the driver profile and preferences. For example, the driver profile and preferences may indicate how the driver typically drives, which may influence the vehicle's mileage.

[0108] The feature-modifying synthesis sensor can use features that are active within the vehicle to examine, among other options, whether the HVAC system is on, whether the seat heaters are on, whether the vehicle windows are open or closed, and can provide recommendations to enable or disable features in order to increase or improve the vehicle's mileage.

[0109] Insights from each of the composite sensors may be provided to the insight broker 492, which may then interact with the mileage application 497, in particular, the user interface 498, using the bridge 496. For example, if the mileage is within a predetermined threshold, the UI may not display any recommendations. However, if the mileage being compared to the destination is within a certain mileage, the user may be offered various options, among other choices, such as recommended driving behavior, enabling or disabling features, or making a discretionary stop to charge the vehicle.

[0110] The insight broker 492 may provide further inputs to return to the trigger service 452, thereby generating a loop for playback. For example, if a driver change is detected, this may trigger a lifecycle service, enabling the feature change synthetic sensor and behavior change synthetic sensor to re-evaluate the insights from such synthetic sensors.

[0111] The embodiment in Figure 4 shows multiple synthetic sensors used to provide a mileage application on a user interface and recommendations on such a mileage application, but this is provided solely for illustrative purposes. In other embodiments, different synthetic sensors and different data inputs using different sensors may also be used, whether other tasks are real-world or simulated.

[0112] (Prototyping)

[0113] As shown above, embodiments of Figures 2, 3, or 4 can be used for rapid prototyping of synthetic sensors. Thus, developers of synthetic sensors can easily plug such synthetic sensors into computing devices such as computing device 430. Data sources can be generated from real-world sensors or simulated sensors. A playback tool 433 can be generated to enable the use of sensor data relating to the synthetic sensor in a lightweight and adaptable manner.

[0114] Specifically, in some embodiments, developers may utilize the abstraction layer either by generating it themselves or by using an existing abstraction layer. However, in other cases, a playback tool such as the playback tool 433 may be used for data conversion from a specific sensor to a format expected by the synthetic sensor. In this case, the developer simply needs to generate a module such as the EV OpenXC module 434, which is used for a specific type of data and can therefore be a lightweight add-on to the simulator 434 for quickly providing data to the synthetic sensor.

[0115] The initial state for testing the synthetic sensor can be set using initial data 414. The initial state may include vehicle settings, but in some embodiments, it may also include the vehicle environment. For example, if the vehicle has been driven on a race track, city road, or highway, this will affect performance and may be set within the initial data 414.

[0116] In one embodiment, the playback tool may be initialized or triggered by the developer to start the data flow, and lifecycle services may be used to coordinate the simulation lifecycle. In some embodiments, a pause state may exist within the playback tool to allow playback to be paused. This may be used, for example, for debugging in some situations. In some cases, the pause state may cause the playback tool to completely terminate playback before all data has been executed.

[0117] In further cases, the start state may allow playback to begin at a specific point in the data stream.

[0118] In some embodiments, the cam-over parameter within the playback tool can be easily adjusted to cam-on by, for example, using an easily accessible variable, user interface, or other means of modifying such parameter, thereby optimizing the performance of various synthetic sensors under prototyping.

[0119] In other embodiments, prototyping may be for an on-vehicle application or user interface, either without or in addition to the synthetic sensor. In this case, the use of various sensors can again be quickly adapted to provide data to the various synthetic sensors using the playback tool 433. In this case, the output or user interface 498 can be monitored, and the simulation can be started or stopped using the pause feature in the playback tool.

[0120] (Demonstration)

[0121] In further cases, the embodiment shown in Figure 2-4 may be used to provide a presentation for an audience. Specifically, a demonstration may need to be provided regarding the use of such a system in order to demonstrate the features of the vehicle system to a client, stakeholder, developer, or other party. For example, a playback may be provided for a road trip, in which a specific driver riding in the vehicle is identified, and a mileage value based on driver behavior and past driving history may be generated, taking into account the features enabled within the vehicle, and the output of the mileage application user interface 498 may be shown to the audience. This may show the identification of the driver and how it affects the outputted mileage.

[0122] Impediments such as traffic volume, low temperature, or other factors that may affect vehicle mileage may be introduced into the data script file, which may then lead to changes and recommendations regarding behavior, or recommendations to enable or disable features, which may be presented to the user on the UI for presentation purposes.

[0123] The pause or stop feature may allow the presenter to pause or stop, for example, by stopping the data flow, through the use of a script within the playback tool, in order to emphasize it among other options, or to answer a question.

[0124] The embodiment shown in Figure 2-4 therefore provides playback of sensor data within a computing system for the development and testing of synthetic sensors. For safety reasons, playback is performed on the computing device, not on the vehicle itself. Playback can be started and stopped. Playback can further process real-world data through the data source.

[0125] Among other options, in particular, multiple formats for playback, such as the OpenXC simulator, the OpenCV simulator, and direct data, may be provided in the embodiment shown in Figure 2-4.

[0126] In some embodiments, the playback tool can be used as a data converter when it is present on a real-world vehicle and an abstraction layer regarding specific sensors has not yet been written.

[0127] In some embodiments, the playback tool can be extended to take into account the environment in which the vehicle is located, such as racetrack versus road, among other options.

[0128] An exemplary method relating to a computing device is shown with respect to Figure 5. In particular, the process in Figure 5 starts in block 510 and proceeds to block 520, where the computing device may optionally initialize the composite sensor using various initialization parameters. The initialization parameters may define, for example, initial settings for various features on the computing device.

[0129] From block 520, the process proceeds to block 530, in which the computing device may receive sensor data. In particular, the reception of sensor data may be from different computers through a communication subsystem on the computing device, from a database within the computing device itself, or from sensors connected to the computing device via a wired or wireless connection, among other options.

[0130] From block 530, the process proceeds to block 540, in which the playback tool may transform the sensor data. For example, the transformation may be to adjust the sampling rate of the data to the sampling rate expected for the synthetic sensor. In other embodiments, the transformation may include changing the data format of the sensor data to the data format expected by the synthetic sensor. In other embodiments, the transformation may include changing the data type. For example, an image file may be transformed into a mathematical matrix. In other embodiments, the transformation of the sensor data may include normalizing the sensor data to the level expected by the synthetic sensor. In other embodiments, the transformation may include one or more of the above.

[0131] Once the sensor data is converted from block 540 to the converted data format, the converted data is then provided in block 550 to at least one synthetic sensor on the computing device.

[0132] This process then proceeds back from block 550 to block 530 and may continue receiving sensor data until playback ends.

[0133] Optionally, a pause function 560 may exist in which the converted data is not provided to the composite sensor until the user resumes the data flow. Furthermore, other functions may exist, such as a start function to resume playback from a midpoint in the route or to initiate playback. In some cases, a stop function can be used to stop playback before all data has been played back. Further options regarding the functionality of using the playback tool are also possible.

[0134] In an optional embodiment, once the synthetic sensor is provided with converted data, it may then provide an output used to provide an output to a display of a computing device.

[0135] The computing devices, platforms, electronic control units, nodes, or other computing systems described above can be implemented using any computing device. A simplified schematic diagram of one computing device is shown with respect to Figure 6. The computing device in Figure 6 may be any fixed or mobile computing device.

[0136] In Figure 6, device 610 includes a processor 620 and a communication subsystem 630, the processor 620 and the communication subsystem 630 cooperating to carry out the methods of the embodiments described above. The communication subsystem 630 enables device 610 to communicate with other devices or network elements and may vary depending on the type of communication to be carried out. Furthermore, the communication subsystem 630 may comprise multiple communication technologies, including any wired or wireless communication technologies.

[0137] The processor 620 is configured to execute programmable logic, which is stored on the device 610 along with data and may be shown as memory 632 in an example in Figure 6. Memory 632 may be any tangible, non-transient, computer-readable storage medium that stores instruction code that causes the device 610 to perform the method of the present disclosure when executed by the processor 620. The computer-readable storage medium may be tangible or transient / non-transient media such as optical (e.g., CD, DVD, etc.), magnetic (e.g., tape), flash drive, hard drive, or other memory known in the art.

[0138] Alternatively, or in addition to memory 632, device 610 may access data or programmable logic from an external storage medium, for example, through the communication subsystem 630.

[0139] In the example in Figure 6, one or more sensors 640 may be associated with the computing device. However, this is optional, and in some cases, the computing device 610 may not be associated with any sensors.

[0140] In one embodiment, communication between the various elements of device 610 may occur via an internal bus 660. However, other forms of communication are also possible.

[0141] The embodiments described herein are examples of structures, systems, or methods having elements corresponding to elements of the technique of the present application. The descriptions provided may enable a person skilled in the art to construct and use embodiments having alternative elements similarly corresponding to elements of the technique of the present application. The intended scope of the technique of the present application therefore includes other structures, systems, or methods that are not different from the technique of the present application as described herein, and further includes other structures, systems, or methods that have non-substantial differences from the technique of the present application as described herein.

[0142] While the actions are depicted in a specific order in the diagrams, this should not be understood as requiring that such actions be performed in a specific or sequential order to achieve the desired result, or that all illustrated actions be performed. In some situations, multitasking and parallel processing may be employed. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together within a single software product, or packaged within multiple software products.

[0143] Techniques, systems, subsystems, and methods described and illustrated in various implementations, either separately or distinctly, may be combined or integrated with other systems, modules, techniques, or methods. Other items shown or discussed as being coupled, directly coupled, or communicating with one another may be indirectly coupled or communicated through some interface, device, or intermediate component, whether electrical, mechanical, or otherwise. Other examples of modifications, substitutions, and alterations are visible to and may be made by those skilled in the art.

[0144] While the above detailed description has shown, explained, and pointed out fundamental novel features of this disclosure applicable to various implementations, it will be understood that various omissions, substitutions, and modifications in the form and details of the illustrated system can be made by those skilled in the art. In addition, the order of the method steps is not implied by the order in which they appear in the invoice.

[0145] When messages are sent to and from electronic devices, such actions may not be immediate to or directly from a server. They may be delivered synchronously or asynchronously from servers or other computing system infrastructure supporting the devices / methods / systems described herein. The aforementioned steps may, in whole or in part, involve synchronous / asynchronous communication to and from devices / infrastructure. Communication from electronic devices may also be to one or more endpoints on a network. These endpoints may be provided by servers, distributed computing systems, stream processors, etc. Content delivery networks (CDNs) may also provide communication to electronic devices. For example, rather than a typical server response, a server may also provision or indicate data for a Content Delivery Network (CDN) to await later download by the electronic device, such as subsequent activity on the electronic device. Thus, data may be sent directly from a server, or other infrastructure such as a distributed infrastructure, or from a CDN, either as part of a system or separately.

[0146] Typically, storage media can include any or any combination of the following: semiconductor memory devices such as dynamic or static random access memory (DRAM or SRAM), erasable and programmable read-only memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), and flash memory; magnetic disks such as fixed, floppy disks, and removable disks; other magnetic media including tapes; optical media such as compact discs (CDs) or digital video discs (DVDs); or other types of storage devices. Note that the instructions discussed above may be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed across a large system potentially having multiple nodes. Such computer-readable or machine-readable storage media or multiple media are considered part of an article (or product). An article or product may refer to any single or multiple manufactured components. The storage medium or multiple mediums may be located either within a machine that executes machine-readable instructions, or at a remote location from which machine-readable instructions can be downloaded over a network for execution.

[0147] Numerous details are provided in the foregoing description to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations from the details discussed above. The attached claims are intended to cover such modifications and variations.

Claims

1. A method in a computing device, wherein the method is In the computing device, sensor data is received from a data source, In the playback tool on the computing device, the sensor data is converted into transformed data, thereby bypassing the abstraction layer in the computing device. The converted data is provided to at least one synthetic sensor on the computing device. Includes, A method wherein each of the at least one synthetic sensor provides insights into the operation of the computing device.

2. The method according to claim 1, wherein providing the converted data includes bypassing the trigger module and lifecycle module for the at least one composite sensor in the computing device.

3. The method according to claim 1, wherein the conversion utilizes a module configured for the received sensor data and further configured for the at least one composite sensor.

4. The method according to claim 3, wherein the module includes a variable for dynamically adjusting the conversion.

5. The method according to claim 3, wherein the module performs at least one of setting a data sampling rate, converting the data format of the sensor data, converting the data type of the sensor data, and normalizing the sensor data.

6. The method according to claim 1, further comprising pausing, stopping, or starting the provision of the converted data by calling at least one of a pause routine, a stop routine, and a start routine in the playback tool.

7. Prior to receiving the aforementioned sensor data, the computing device receives initialization data, Initialize the at least one composite sensor based on the initialization data. The method according to claim 1, further comprising:

8. The method according to claim 1, further comprising utilizing the insights to update the display associated with the computing device.

9. The method according to claim 1, wherein the sensor data represents data from one or more vehicle sensors, and the computing device virtualizes at least one operating system from a vehicle computing system.

10. A computing device, wherein the computing device is Processor and Communication subsystem and Equipped with, The computing device is Receiving sensor data from the data source, In the playback tool on the computing device, the sensor data is converted into transformed data, thereby bypassing the abstraction layer in the computing device. The converted data is provided to at least one synthetic sensor on the computing device. It is configured to do the following: Each of the at least one synthetic sensor provides insights into the operation of the computing device.

11. The computing device according to claim 10, wherein the computing device is configured to provide the converted data by bypassing the trigger module and lifecycle module for the at least one composite sensor in the computing device.

12. The computing device according to claim 10, wherein the computing device is configured to convert the received sensor data using a module further configured for the at least one composite sensor.

13. The computing device according to claim 12, wherein the module includes a variable for dynamically adjusting the conversion.

14. The computing device according to claim 12, wherein the module performs at least one of setting a data sampling rate, converting the data format of the sensor data, converting the data type of the sensor data, and normalizing the sensor data.

15. The computing device according to claim 10, further configured to pause, stop, or start providing the converted data by calling at least one of a pause routine, a stop routine, and a start routine in the playback tool.

16. The computing device is Prior to receiving the aforementioned sensor data, the computing device receives initialization data, Initialize the at least one composite sensor based on the initialization data. The computing device according to claim 10, further configured to perform the following:

17. The computing device according to claim 10, further configured to utilize the insights to update a display associated with the computing device.

18. The computing device according to claim 10, wherein the sensor data represents data from one or more vehicle sensors, and the computing device virtualizes at least one operating system from a vehicle computing system.

19. A computer-readable medium for storing instruction codes, wherein the instruction codes are executed by the processor of a computing device. Receiving sensor data from the data source, In the playback tool on the computing device, the sensor data is converted into transformed data, thereby bypassing the abstraction layer in the computing device. The converted data is provided to at least one synthetic sensor on the computing device. The computing device is made to perform this task. Each of the at least one synthetic sensor is a computer-readable medium that provides insights into the operation of the computing device.

20. The computer-readable medium according to claim 19, wherein the instruction code is further configured to cause the computing device to provide the converted data to the computing device by bypassing the trigger module and lifecycle module for the at least one composite sensor.

Images