Onboard synthetic sensor orchestration and remote synthetic sensor services
By using synthetic sensor services and orchestration environments, synthetic sensors can be remotely deployed, solving the problem of difficult sensor data sharing between different operating system domains in vehicles. This enables flexible sharing of sensor data and deployment of new functions, simplifies the update process, and improves the scalability and security of vehicle functions.
Patent Information
- Application Number
- CN202180021404.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-03-30
- Filing Date
- 2021-03-19
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-03-19
AI Technical Summary
Sharing sensor data between different operating system domains in existing vehicles is difficult, and changing or updating fixed algorithms and rules is cumbersome and time-consuming, making it difficult to deploy new functions or features.
Synthetic sensors can be remotely deployed through synthetic sensor services and orchestration environments. The synthetic sensor orchestration environment determines the placement and configuration of sensors, allows sensor data to be shared between different domains, and enables flexible sensor logic and mapping through machine learning models and annotation generators.
It enables flexible sharing of sensor data and deployment of new functions in vehicles without changing existing hardware or downgrading certification levels, simplifies the updating and expansion of sensor data, and improves the scalability and security of vehicle functions.
Smart Images

Figure CN115280278B_ABST
Abstract
Description
Background Technology
[0001] Modern vehicles, such as cars, trucks, and motorcycles, are typically manufactured with electronic sensors and computer systems programmed with control algorithms. These algorithms take input from the electronic sensors to determine various control actions to be taken on the vehicle or systems implemented within it. To ensure a high level of safety, these sensors and control algorithms undergo rigorous testing and certification to meet different performance levels, such as the Automotive Safety Integrity Level (ASIL). For example, depending on the criticality of the sensor or algorithm, a higher or lower certification level may be required.
[0002] Some vehicles may contain multiple operating systems or an operating system with multiple domains, where different operating systems or domains are certified at different corresponding performance levels. For example, in some vehicles, radio may be part of an entertainment operating system domain that may not need to meet ASIL certification requirements, while sensors for seat belts and airbags may be included in a safety operating system domain that requires a higher ASIL certification level. In such vehicles, different operating systems / operating system domains may be independent of each other, or may share only a limited amount of data between operating systems / domains. For example, electronic sensor information related to seat belts may not be accessible in the entertainment operating system domain, and vice versa; radio volume levels may not be accessible in the safety operating system domain.
[0003] In addition, vehicles are typically manufactured with fixed control algorithms and rules for sharing electronic sensor data between operating systems or domains, and changing or updating these fixed algorithms and / or rules for sharing electronic sensor data is a cumbersome or time-consuming task. Attached Figure Description
[0004] Figure 1 A synthetic sensor service according to some embodiments is illustrated, which receives instructions from a client of the synthetic sensor service for deploying synthetic sensors and remotely causes the synthetic sensors to be deployed to vehicles, each containing an on-board synthetic sensor orchestration environment.
[0005] Figure 2 A more detailed view of a synthetic sensor package that can be deployed from a synthetic sensor service to an in-vehicle synthetic sensor orchestration environment, according to some embodiments, is shown, wherein the synthetic sensor package contains code logic and mappings within an envelope of the synthetic sensor package, and also includes annotations of the synthetic sensors outside the scope of the envelope.
[0006] Figure 3 Synthetic sensor services, according to some embodiments, are illustrated, configured to deploy a common synthetic sensor package to different types of vehicles, each implementing an onboard synthetic sensor orchestration environment.
[0007] Figure 4 An example provider network is shown, which includes synthetic sensor services and other cloud services provided by the provider network, according to some embodiments.
[0008] Figure 5 An example deployment of a synthetic sensor package according to some embodiments is shown.
[0009] Figure 6 A sample client console for a synthetic sensor service, according to some embodiments, is shown.
[0010] Figure 7 An example view of a synthetic sensor orchestration environment according to some embodiments is shown.
[0011] Figure 8 An example view of a specific synthetic sensor is shown, implemented according to some embodiments, in a synthetic sensor orchestration environment running on top of an in-vehicle operating system.
[0012] Figure 9 An example view of a specific synthetic sensor is shown, implemented according to some embodiments, in a synthetic sensor orchestration environment running on top of a vehicle's entertainment operating system.
[0013] Figure 10 An example view of a specific synthetic sensor is shown, according to some embodiments, in a synthetic sensor orchestration environment running in a gateway between the vehicle's operating system and the vehicle's infotainment operating system.
[0014] Figure 11 A flowchart illustrating the operation of a synthetic sensor service according to some embodiments is shown.
[0015] Figure 12 A flowchart illustrating the operation of an onboard synthetic sensor orchestration environment according to some embodiments is shown.
[0016] Figure 13 A method for reconfiguring synthetic sensor placement in two or more operating systems or operating system domains in response to changes in the vehicle's operating system or operating system domain, according to some embodiments, is illustrated.
[0017] Figure 14 This is a block diagram illustrating an example computer system implementing some or all of the techniques described herein according to some embodiments.
[0018] Although embodiments have been described herein by way of example with respect to several embodiments and illustrative drawings, those skilled in the art will recognize that the embodiments are not limited to the described embodiments or drawings. It should be understood that the drawings and detailed descriptions thereof are not intended to limit the embodiments to the specific forms disclosed, but rather are intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope defined by the appended claims. The headings used herein are for organizational purposes only and are not intended to limit the scope of the description or claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning the possibility of…) rather than a mandatory sense (i.e., meaning the requirement). Similarly, the words “include,” “including,” and “includes” mean including but not limited to. Detailed Implementation
[0019] The systems and methods described herein include techniques for implementing synthetic sensor services and implementing in-vehicle synthetic sensor orchestration environments, enabling clients of the synthetic sensor service to deploy synthetic sensors in vehicles already available to vehicle operators. For example, vehicle manufacturers, vehicle parts manufacturers, or other third parties can develop new applications using new types of synthetic sensors and deploy these new applications and / or synthetic sensors in vehicles already sold to consumers. Deployment of new types of synthetic sensors can be achieved remotely via a network connection between the synthetic sensor service and the vehicle, wherein the vehicle's onboard computing device implements a synthetic sensor orchestration environment that determines the placement of the new synthetic sensor and further performs configuration operations to enable the transmission of required input data from existing physical sensors or from other synthetic sensors in the vehicle to the new synthetic sensor. The synthetic sensor orchestration environment also performs configuration operations to enable the transmission of output data from the new synthetic sensor to its output destination.
[0020] In some embodiments, synthetic sensors are deployed to vehicles already available to vehicle operators, either in addition to or instead of using synthetic sensor services. For example, clients such as vehicle manufacturers may use synthetic sensor services to deploy synthetic sensors to vehicles during the vehicle manufacturing process. For instance, during a portion of the vehicle manufacturing process, after the onboard computing system has been installed and connected to the vehicle's communication bus, synthetic sensors may be deployed to a synthetic sensor orchestration environment to perform quality assurance testing on the vehicle's electronic systems and / or provide output data from the synthetic sensors for use during vehicle manufacturing and testing.
[0021] In some embodiments, a system includes one or more computing devices storing program instructions for implementing synthetic sensor services for a vehicle. When executed by the one or more computing devices, the program instructions cause the one or more computing devices to implement an interface to receive instructions for deploying one or more synthetic sensors to the vehicle during the manufacturing process or after the vehicle is available to the operator of the vehicle. When executed by the one or more computing devices, the program instructions further cause the one or more computing devices to receive, via the interface, a selection of one or more synthetic sensors to be deployed to the vehicle, wherein the interface, for the selected one or more synthetic sensors, instructs mapping the one or more synthetic sensors to one or more existing physical sensors or other synthetic sensors of the vehicle. Additionally, when executed by the one or more computing devices, the program instructions cause the one or more computing devices to provide one or more selected synthetic sensor packages for a synthetic sensor orchestration environment for deployment to the vehicle based on the received selection, wherein the provided one or more synthetic sensor packages include code defining logical elements and mappings for the corresponding one or more synthetic sensors and one or more corresponding annotations defining the inputs, outputs, or attributes of the corresponding one or more synthetic sensors, wherein the code is included in an encapsulated format along with one or more annotations outside the scope of the code envelope.
[0022] In some embodiments, a system includes one or more computing devices configured to be installed in a vehicle, wherein the one or more computing devices store program instructions for implementing a synthetic sensor orchestration environment in the vehicle. The synthetic sensor orchestration environment is configured to receive synthetic sensor packets from a synthetic sensor service, the synthetic sensor packets including code defining logic for synthetic sensors and one or more annotations defining inputs, outputs, or attributes of the synthetic sensors. The code for synthetic sensors is included in an encapsulated format along with one or more annotations outside the scope of the code envelope. The synthetic sensor orchestration environment is also configured to determine a specific operating system domain for implementing the synthetic sensors based on one or more annotations. Additionally, the synthetic sensor orchestration environment is configured to perform one or more configuration operations to enable the synthetic sensors to access input data from physical sensors or other synthetic sensors within the synthetic sensor orchestration environment.
[0023] In some embodiments, one or more non-transitory computer-readable storage media store program instructions that, when executed on or across one or more computing devices, cause one or more computing devices to receive a synthetic sensor package. The synthetic sensor package includes code defining logic for a synthetic sensor and one or more annotations defining inputs, outputs, or attributes of the synthetic sensor, wherein the code is included in an encapsulated format along with one or more annotations outside the scope of the code envelope. The program instructions also cause one or more computing devices to determine a specific operating system domain for implementing the synthetic sensor based on the one or more annotations. Additionally, the program instructions cause one or more computing devices to perform one or more configuration operations to enable the synthetic sensor to access input data from one or more physical sensors or synthetic sensors of a vehicle contained within the specific operating system domain.
[0024] Vehicles are typically manufactured with electronic sensors and computing devices that execute control algorithms, which take sensor data from the electronic sensors as input. However, the communication interfaces of the electronic sensors used in vehicles often differ between different vehicle manufacturers, and even between different models manufactured by the same manufacturer. Furthermore, electronic sensors and associated control algorithms are often implemented as independent systems, serving a specific purpose for the vehicle and not configured to share sensor data with other systems within the vehicle. For example, a vehicle's entertainment system may be implemented using electronic sensors (e.g., volume controls), but may not be configured to share sensor data with other systems in the vehicle. In some cases, this is done because independent systems within the vehicle have different certification requirements, allowing each system to remain independent and be certified separately based on independent hardware and control algorithm requirements.
[0025] Furthermore, vehicles are typically manufactured with fixed algorithms and rules for sharing sensor data, making changes to these fixed algorithms or rules a laborious and / or time-consuming task. For example, the algorithms and rules for sharing data are usually determined during the vehicle's design phase and cannot be changed after the vehicle has been manufactured, or may require physical replacement of parts such as computing devices to change the fixed algorithms or rules for sharing sensor data.
[0026] For example, changing the algorithms or rules used to share sensor data might require vehicle owners to take their vehicles to a service center and physically replace a vehicle computer programmed with a specific set of algorithms and / or rules for sharing sensor data with another vehicle computer that has a different configuration. These obstacles can hinder the deployment of new applications and sensors in vehicles that utilize relationships between sensor data already collected by electronic sensors installed on the vehicle to determine new functionality or features of the vehicle.
[0027] For example, a vehicle can be manufactured with seat weight sensors useful for determining whether an occupant is sitting in a vehicle seat. The seat weight sensor can be included in the vehicle's safety domain and can be used in an alarm to remind the occupant to fasten their seatbelt. Additionally, the vehicle can include a cockpit camera, which serves as another layer within the safety domain to detect driver distraction or fatigue. However, the different layers of the safety domain can be independent, so that the two domains do not share information. This may be because the certification level for the distraction / fatigue driving layer is lower than the certification level for the seatbelt layer of the safety domain, where different certification levels require different types of hardware components to implement the programming algorithms for the corresponding domains. However, OEM manufacturers, OEM parts manufacturers, or other third parties can determine that sensor data from the seat weight sensor and the cockpit camera can be combined to determine whether a child is sitting in the seat. Furthermore, other applications can use the fact that a child is sitting in the seat to improve functionality. For example, an entertainment system can customize entertainment options available to the seat occupant based on the knowledge that a child, rather than an adult, is sitting in the seat.
[0028] While the infotainment and safety domains may not be configured separately to provide this functionality, as described herein, a synthetic sensor orchestration environment enables OEMs, OEM parts manufacturers, or third parties to deploy “Child in seat?” synthetic sensors within the environment. This environment allows and / or disallows sensor data from existing physical sensors in the same or different domains to be passed to the new synthetic sensor, and further allows the addition of new logic code for the new synthetic sensor that uses sensor data from the existing physical sensors to determine new outputs, such as “Child in seat, yes or no?” outputs. In this example, the synthetic sensor output can then be used as input to an infotainment system that customizes content provided to the occupant in the seat based on the occupant’s understanding of the child. While this is a specific example of the use of synthetic sensors implemented within a synthetic sensor orchestration environment, various other inputs from various other types of existing physical sensors can be used as inputs to synthetic sensors to determine new types of outputs that can be used to provide new features in a vehicle.
[0029] As another example, a synthetic sensor service may provide one or more synthetic sensor packages to a vehicle during the vehicle's manufacturing process, after one or more computing devices storing program instructions for implementing a synthetic sensor orchestration environment are installed in the vehicle. These one or more synthetic sensors may be configured to generate outputs for testing the vehicle during the manufacturing process. For example, the synthetic sensors may test other components of the vehicle to determine the proper installation and configuration of those components within the vehicle.
[0030] Furthermore, this synthetic sensor orchestration environment allows new synthetic sensors to be easily added to vehicles via network-connected deployment. For example, a synthetic sensor package can be deployed within the synthetic sensor orchestration environment without requiring any physical hardware replacement in the vehicle or taking the vehicle to a service center for reprogramming. Additionally, the synthetic sensor orchestration environment can be situated on top of existing operating system domains, such as vehicle control domains, safety domains, infotainment domains, etc., allowing the synthetic sensor orchestration environment to receive sensor data from sensors in the corresponding domain without altering the code logic or algorithms within that domain, and without otherwise degrading the certification level of the corresponding domain and the electronic sensors within it. For example, adding a "Child in seat, yes or no?" synthetic sensor may not change or degrade the seat weight sensor or the algorithm in the safety domain that uses the seat weight sensor to indicate a seatbelt alarm.
[0031] Furthermore, in some embodiments, the synthetic sensor service can provide a unified deployment mechanism for synthetic sensors in vehicles with different electronic sensor interfaces. For example, synthetic sensor orchestration environments in different types of vehicles may include interfaces that allow or disallow sensor data from physical sensors in different types of vehicles, but can be configured to receive common synthetic sensor packages deployed from the synthetic sensor service. For example, a first vehicle manufacturer and a second vehicle manufacturer may use different types of seat weight sensors with different communication interfaces. However, the synthetic sensor orchestration environment in each of the different types of vehicles can be configured to receive a common synthetic sensor package for a “child is sitting in the seat, yes or no?” synthetic sensor, and locally determine the placement of the synthetic sensor and the configuration operations that need to be taken to allow data from existing sensors to be transmitted to the new synthetic sensor. Thus, a third-party developer can develop a single version of a “child is sitting in the seat, yes or no?” synthetic sensor and use the synthetic sensor service to deploy a new “child is sitting in the seat, yes or no?” synthetic sensor to synthetic sensor orchestration environments in multiple different types of vehicles with different sensor communication interfaces, wherein the placement in the respective synthetic sensor orchestration environment is determined locally.
[0032] Additionally, in some embodiments, the synthetic sensor package deployed from the synthetic sensor service to the synthetic sensor orchestration environment in the vehicle may include one or more annotations that extend beyond the envelope of the synthetic sensor package, which encapsulates the control logic and mapping of the new synthetic sensor. Such annotations can be used by the synthetic sensor orchestration environment to determine the placement of the new synthetic sensor. Furthermore, these annotations can be used by the synthetic sensor orchestration environment to further determine the impact of the placement of the new synthetic sensor in a particular system domain on other synthetic sensors. For example, the newly placed synthetic sensor may provide outputs listed as optional inputs in the annotations of another synthetic sensor, where the presence of optional inputs allows the other synthetic sensor to be upgraded to a higher certification level or otherwise provide more reliable outputs.
[0033] For example, annotations can be included as metadata in the synthetic sensor package, outside the scope of the code (e.g., logical elements and mappings) of the new synthetic sensor itself. Therefore, the synthetic sensor orchestration environment can read the annotations contained in the metadata of the synthetic sensor package to determine the placement decision of the new synthetic sensor, without having to parse the code (e.g., logical elements and mappings) of the new synthetic sensor contained within the envelope to determine the placement decision.
[0034] In some embodiments, the synthetic sensor may include code that uses sensor data from existing sensors to determine the output of the new synthetic sensor. Furthermore, in some embodiments, the synthetic sensor may include one or more machine learning models that use sensor data from existing sensors to determine the output of the new synthetic sensor. For example, the machine learning model may be a logical element of the synthetic sensor, wherein physical sensor inputs or other inputs are mapped to the machine learning model, and the output of the machine learning model is further used by the synthetic sensor to determine the output of the synthetic sensor.
[0035] In some embodiments, OEM manufacturers, OEM parts manufacturers, and / or third-party developers may develop new applications for vehicles using new types of synthetic sensors. In such embodiments, clients may purchase new applications from app stores, and the app stores may advise the vehicle's OEM to instruct the synthetic sensor service to deploy one or more synthetic sensor packages to the client's vehicle to implement the purchased new application. In some embodiments, the synthetic sensor service and corresponding synthetic sensor orchestration environments in different vehicles can provide a consistent view for controlling sensor data for use in updated code after the vehicle has been manufactured and / or put into use by consumers. For example, a synthetic sensor orchestration environment may enable sensor data to be shared horizontally throughout the vehicle, rather than being limited to multiple separate vertically oriented control domains.
[0036] For example, an OEM manufacturer, OEM parts manufacturer, or another third party can use the synthetic sensor service's console to develop new synthetic sensors. The synthetic sensor developer (e.g., an OEM manufacturer, OEM parts manufacturer, or third party) can select sensor data from a list of sensor data types obtained from existing sensors to use as input to the synthetic sensor. The developer can also select logical elements (e.g., algorithms (e.g., rules) or models (e.g., machine learning models)) to receive the selected input. The developer can further determine the mapping between the input, logical elements (e.g., rules and / or models), and the output of the new synthetic sensor. In some embodiments, the mapping can be represented as a JSON object or XML code, wherein the synthetic sensor orchestration environment to which the new synthetic sensor package is deployed in the vehicle parses the JSON object or XML code to determine the relationship between the input, logical elements (e.g., rules and / or models), and output. The synthetic sensor orchestration environment can then cause one or more configuration operations to implement the logical elements (e.g., rules and / or models) mapped to the code logic of the synthetic sensor package. The synthetic sensor orchestration environment can further cause one or more additional configuration operations to allow sensor data to be passed to components of the synthetic sensor according to the mapping contained in the code logic of the synthetic sensor package. In some embodiments, logical elements and mappings may be contained within an encapsulated environment, and one or more annotations stored as metadata of the synthetic sensor package (outside the envelope) may indicate characteristics of the synthetic sensor, such as required inputs, optional inputs, outputs, authentication levels, synthetic sensor dependencies, failure modes, etc.
[0037] In some embodiments, an OEM manufacturer, OEM parts manufacturer, or another third party may use a service interface that includes a console or is separate from the synthetic sensor design console to instruct the synthetic sensor service to deploy one or more synthetic sensors to a specific vehicle or class of vehicles eligible to receive synthetic sensors. For example, an OEM manufacturer, OEM parts manufacturer, or another third party may require consumers to purchase an upgrade package to access new types of features enabled by using a new type of synthetic sensor.
[0038] In some embodiments, synthetic sensor sets can be grouped together to perform more complex functions. For example, in some embodiments, the output of a first synthetic sensor can be the input of another synthetic sensor. Furthermore, in some embodiments, a synthetic sensor orchestration environment in a vehicle can be configured to share at least some vehicle sensor data via a network connection and storage service, or via a network connection between the vehicle and a provider network and another cloud service of the provider network. For example, sensor data from the vehicle can be provided to a machine learning service of the provider network to determine improved control relationships, where the results of the machine learning are subsequently used to implement new functions or synthetic sensors in the vehicle. In some embodiments, machine learning can be used in conjunction with the vehicle's control algorithms, or it can be used to determine control relationships that are subsequently translated into static, auditable code to be deployed to the vehicle. Furthermore, in some embodiments, a client, such as an OEM, can operate a data storage facility, and the synthetic sensor orchestration environment in the vehicle can be configured to share at least some vehicle sensor data with the OEM's data storage facility via a network connection.
[0039] In some embodiments, the synthetic sensor may be developed and controlled by the OEM manufacturer and / or OEM parts manufacturer to ensure that the synthetic sensor deployed in vehicles manufactured by the OEM or OEM parts manufacturer meets specific quality and / or certification requirements. In other embodiments, other third parties may be able to develop new types of synthetic sensors and deploy them in vehicles manufactured by another party (e.g., an OEM or multiple different OEMs).
[0040] In some embodiments, the various domains or operating systems included in the vehicle may include an infotainment domain / OS, a cockpit or control domain / OS, a communications domain / OS, a security system domain / OS, a vehicle server domain / OS, a telematics communication unit domain / OS, an advanced driver assistance system domain / OS, a cloud domain / OS, an edge processing domain / OS (which may be partially implemented in cellular communication towers, etc.), and / or a gateway domain / OS. The vehicle may include a common communication bus, but different domains may be independent branches of the bus, or data transmitted via the bus may not be accessible in all domains.
[0041] In some embodiments, a synthetic sensor orchestration environment may provide the output from a synthetic sensor to another vehicle system outside the range of the synthetic sensor orchestration environment. In some embodiments, the other vehicle system may take one or more control actions based on the output received from the synthetic sensor. For example, if the synthetic sensor is one that determines whether a passenger is sitting in the front seat, the output of the synthetic sensor may be provided to a vehicle system outside the range of the synthetic sensor orchestration environment, such as a front seat adjustment mechanism controller, which automatically moves the front seat forward when no one is sitting in the front seat, thereby increasing the legroom available to rear passengers.
[0042] In some embodiments, the synthetic sensor orchestration environment can provide the output from the synthetic sensors to a remote system outside the vehicle's range. For example, a portion of the vehicle's infotainment system can be implemented remotely using cloud-based resources, and the output of the synthetic sensors can be provided to a cloud provider outside the vehicle's range. For instance, the output of the "Children in the vehicle?" synthetic sensor can be provided to a remote server that customizes media content to be provided to the vehicle, where different customizations of the media content can be performed based on whether or not children are in the vehicle.
[0043] Figure 1 A synthetic sensor service according to some embodiments is illustrated, which receives instructions from a client of the synthetic sensor service for deploying synthetic sensors and remotely causes the synthetic sensors to be deployed to vehicles, each containing an on-board synthetic sensor orchestration environment.
[0044] In some embodiments, a synthetic sensor service for vehicles, such as synthetic sensor service 102, is configured to receive instructions for defining new synthetic sensors from clients, such as clients 120a to 120n, via a network, such as network 141. In some embodiments, clients 120a to 120n may include vehicle original equipment manufacturers (e.g., vehicle OEMs), vehicle parts original equipment manufacturers (e.g., parts OEMs), or third parties (e.g., application developers).
[0045] In some embodiments, the synthetic sensor service 102 may generate a synthetic sensor package 146 based on instructions received from any of clients 120a to 120n, and deploy the synthetic sensor package to a synthetic sensor orchestration environment 140 of one or more vehicles (e.g., vehicles 124, 148, 150, etc.) via network 122. For example, network 122 may be a wireless network, such as a cellular, satellite, or other network that allows data communication between the synthetic sensor service 102 and an onboard computing system 152 implementing the synthetic sensor orchestration environment 140.
[0046] In some embodiments, a synthetic sensor service, such as synthetic sensor service 102, may include a code logic repository 104, a physical sensor list 106, a service interface 108, a mapping and annotation generator 110, an application programming interface 112, and a vehicle communication interface 114.
[0047] In some embodiments, such as service interface 108, the service interface of a synthetic sensor service may implement a service console (e.g., Figure 6 (As shown), an application programming interface, command-line interface, graphical user interface, or other suitable interface configured to receive synthetic sensor instructions from clients 120a to 120n, or enabling clients to select and / or design new synthetic sensors to be deployed to a vehicle containing a synthetic sensor orchestration environment. It should be noted that in some embodiments, vehicles such as vehicle 124 may be manufactured to include a synthetic sensor orchestration environment 140 at manufacturing time, wherein the synthetic sensor orchestration environment 140 enables new synthetic sensors to be deployed to vehicle 124 after manufacturing and after being put into use by the owner or operator of vehicle 124.
[0048] In some embodiments, the console implemented by service interface 108 can provide clients 120 with available sensors contained in vehicles for designing new synthetic sensors. For example, physical sensor list 106 can store an updated list of available sensors in various types of vehicles subscribed to synthetic sensor service 102. Furthermore, the console implemented by service interface 108 can provide clients with available rules, models, etc., for designing synthetic sensors. For example, code logic repository 104 can store logical elements, such as available rules, models, etc., that can be used to define the code logic of new synthetic sensors. Additionally, in some embodiments, clients can submit client-generated rules, models, etc., to be used in the new synthetic sensor. In some embodiments, service interface 108 can provide a drag-and-drop graphical user interface (GUI) where clients drag and drop logical elements (e.g., rule models, etc.) to be used in the new synthetic sensor, and further define the data flow through the new synthetic sensor by drawing lines that connect inputs to logical elements and further connect logical elements to one or more outputs of the synthetic sensor. In some embodiments, the client can select logical elements from predefined logical elements for drag-and-drop to define a new synthetic sensor, or the client can provide client-defined logical elements for drag-and-drop and further connection with other code elements, inputs, outputs, etc. to define a new synthetic sensor.
[0049] In some embodiments, the mapping and annotation generator 110 may then generate a synthetic sensor package defined by the client via service interface 108. In some embodiments, the client may specify annotations to be included with the synthetic sensor package, or the mapping and annotation generator 110 may automatically determine the annotations based on the synthetic sensor definition received via service interface 108.
[0050] In some embodiments, the application may request the deployment of a synthetic sensor to enable the application to be implemented in a vehicle, and the application interface 112 may receive this request.
[0051] In some embodiments, the vehicle communication interface 114 may establish a network connection with the on-board computing device 152 and may transmit the synthetic sensor package 146 to the synthetic sensor orchestration environment 140 for deployment.
[0052] In some embodiments, a vehicle, such as any of vehicles 124, 148, 150, etc., may include one or more in-vehicle computing devices 152 implementing one or more operating system domains (e.g., operating system domains 136 to 138). Additionally, the in-vehicle computing device 152 may implement a synthetic sensor orchestration environment 140 spanning one or more operating system domains (e.g., operating system domains 136 and 138). In some embodiments, each operating system domain may be implemented on a separate in-vehicle computing device or on a common in-vehicle computing device. Furthermore, in some embodiments, the synthetic sensor orchestration environment 140 may be implemented on an in-vehicle computing device sharing the same one or more operating system domains, or on a separate in-vehicle computing device that interfaces with other in-vehicle computing devices implementing the respective operating system domains 136 to 138. Furthermore, in some embodiments, the vehicle may include a communication bus 132 connecting the in-vehicle computing device 152 to physical sensors (e.g., physical sensors 126, 128, and 130). In some embodiments, one or more different types of interfaces 134 may connect the physical sensors to the communication bus 132. Furthermore, in some embodiments, the synthetic sensor orchestration environment 140 may implement one or more synthetic sensors, such as synthetic sensors 142 and 144. In some embodiments, the synthetic sensors may be implemented in a specific operating system domain or may be permitted to communicate with other parts of the vehicle via a communication bus (e.g., communication bus 132).
[0053] Figure 2 A more detailed view of a synthetic sensor package that can be deployed from a synthetic sensor service to an in-vehicle synthetic sensor orchestration environment, according to some embodiments, is shown, wherein the synthetic sensor package contains code logic and mappings within an envelope of the synthetic sensor package, and also includes annotations of the synthetic sensors outside the scope of the envelope.
[0054] In some embodiments, a synthetic sensor package, for example Figure 1 The synthetic sensor package 146 described herein may contain, with Figure 2 The format is similar to that of the synthetic sensor package 202 described in the document.
[0055] Synthetic sensor package 202 includes an envelope 204 containing code that encapsulates the definition logic and mapping 206. Furthermore, outside the scope of envelope 204, synthetic sensor package 202 includes annotations 208. In some embodiments, the annotations may include required inputs for the synthetic sensor, optional inputs for the synthetic sensor, certification levels for the synthetic sensor, contingency plans for the synthetic sensor, dependencies of the synthetic sensor, etc. In some embodiments, the contingency plan may indicate alternative inputs for the synthetic sensor to use in cases where a particular required or optional input is unavailable. Furthermore, certification levels and / or contingency plans may indicate different certification levels assigned to the synthetic sensor based on the number of required and / or optional inputs available. Moreover, the certification level of the synthetic sensor may depend on the domain in which the synthetic sensor is implemented. For example, hardware used in a security domain may be more robust than hardware used in an entertainment domain. Therefore, to achieve a higher level of certification, it may be necessary to implement the synthetic sensor in a domain containing more robust hardware that meets the requirements of a given certification level.
[0056] In some embodiments, it may have been obtained from, for example Figure 1 The code logic repository 104 described herein selects logical elements such as rules A, B, C, and D, and models A and B. Furthermore, in some embodiments, one or more of rules A, B, C, or D, or models A or B, may have been provided by the client 120. Mapping 210 can define the required and optional inputs, outputs, and data flows between rules A, B, C, and D, and models A and B. In some embodiments, the client can use service interface 108 to draw lines for mapping 210 to define the synthetic sensor type.
[0057] In some embodiments, note 208 may additionally indicate that the synthetic sensor output is an input to another type of synthetic sensor and / or may indicate that the output from another type of synthetic sensor is a required or optional input to a given synthetic sensor. In some embodiments, the note may indicate the specific operating system domain in which the synthetic sensor is to be deployed. For example, to achieve a specific certification level, it may be necessary to deploy the synthetic sensor in an operating system domain with hardware or processing that meets the requirements of the specific certification level.
[0058] Furthermore, annotations such as Note 208 can indicate that a given synthetic sensor can be upgraded to a higher certification level when one or more optional inputs are available. In some embodiments, when a new synthetic sensor is added, the synthetic sensor orchestration service can scan the annotations of implemented synthetic sensors to determine whether the new synthetic sensor is an optional input to one or more implemented synthetic sensors. Furthermore, the synthetic sensor orchestration environment can automatically change the placement of one or more synthetic sensors (e.g., reconfigure synthetic sensors) so that a given synthetic sensor in the synthetic sensor suite can be upgraded to a higher certification level by utilizing the output of the newly added synthetic sensor as an input to one or more implemented synthetic sensors. In some embodiments, upgrading the certification level of a given synthetic sensor can further allow upgrading other synthetic sensors by receiving inputs that conform to the higher certification level (from the output of the upgraded synthetic sensor).
[0059] Figure 3 Synthetic sensor services, according to some embodiments, are illustrated, configured to deploy a common synthetic sensor package to different types of vehicles, each implementing an onboard synthetic sensor orchestration environment.
[0060] In some embodiments, the input interface to the synthetic sensor orchestration environment can be standardized so that the synthetic sensor orchestration environment can receive a common synthetic sensor package from a synthetic sensor service, while also interfacing with different communication interfaces of electronic sensors contained in different types of vehicles. For example, synthetic sensor service 102 can deploy the common synthetic sensor package 308 to each of several different types of vehicles, such as sedan 302, truck 304, and van 306, which may be manufactured by different vehicle manufacturers and may contain electronic sensors manufactured by different parts manufacturers.
[0061] Figure 4 An example provider network is shown, which includes synthetic sensor services and other cloud services provided by the provider network, according to some embodiments.
[0062] In some embodiments, a provider network, such as provider network 402, includes a networked device 404, a computing device 406, and a data storage device 408 implementing cloud service 410. In some embodiments, the provider network may implement multiple cloud services in addition to the synthetic sensor service. For example, provider network 402 implements cloud service 410, which includes IoT software update service 412, computing service 414, data storage service 416, machine learning service 418, workflow service 420, and other services 422. Cloud service 410 also includes synthetic sensor service 102.
[0063] In some embodiments, an IoT software update service, such as IoT software update service 412, can facilitate devices connected to the IoT software update service (e.g., such as...). Figure 1 Vehicles 124, 148, and 150 shown, or as Figure 3 Software updates on vehicles 302, 304, and 306 shown in the figures. In some embodiments, the IoT software update service may additionally update the firmware on connected devices and may utilize encrypted communication to perform the updates. In some embodiments, the IoT software update service may also include an authentication protocol to prevent unauthorized entities from altering the software on connected devices, such as vehicles, and may encrypt communication with connected devices to prevent changes to the software updates.
[0064] In some embodiments, computing services such as computing service 414 may include computing devices that implement virtual computing machines, which can be used to analyze collected vehicle usage information and / or to implement synthetic sensor services.
[0065] In some embodiments, a data storage service, such as data storage service 416, may include a data storage device that implements virtualized data storage, such as a virtual data storage volume or virtual storage container for object storage. In some embodiments, a data storage service, such as data storage service 416, may be used to store collected vehicle usage information for clients. Furthermore, in some embodiments, a data storage service, such as data storage service 416, may be used to implement various components of a synthetic sensor service. For example, in some embodiments, the code logic repository 104 and the physical sensor list 106 may be implemented using the virtual storage resources of data storage service 416.
[0066] In some embodiments, a machine learning service may execute one or more machine learning algorithms to determine relationships to be used in a synthetic sensor, optimize a workflow service, or optimize another service. For example, in some embodiments, a machine learning service such as machine learning service 418 may be used to optimize the software of a synthetic sensor and may utilize collected vehicle usage information from a synthetic sensor orchestration environment such as synthetic sensor orchestration environment 140 to perform such machine learning or generate machine learning inference outputs.
[0067] In some embodiments, a workflow service, such as workflow service 420, may execute a workflow based on input data and stored or developed workflows. For example, in some embodiments, a workflow service, such as workflow service 420, may determine the action to be taken based on collected vehicle usage information. For example, a workflow service may determine that a filter, hose, or fluid needs to be replaced or replaced based on applying the collected vehicle usage information to one or more stored or developed workflows. In some embodiments, a machine learning service, such as machine learning service 418, may be used to develop or improve workflows executed by workflow service 420.
[0068] In some embodiments, cloud service 410 may include various other cloud services, such as other services 422.
[0069] In some embodiments, the synthetic sensor service may be configured to establish a secure connection with a client network for delivering data from the vehicle synthetic sensor service. For example, in some embodiments, an OEM may receive certain types of vehicle data for improving vehicle design and / or for developing new synthetic sensors. In some embodiments, a client network, such as an OEM network, may establish a secure connection with the synthetic sensor service via a direct connection, such as direct connection 452, or via a logically isolated network connection, such as logically isolated network connection 454. In some embodiments, a direct connection may include a dedicated physical network link linking the client network 450 to the provider network 402 and / or one or more logically isolated network connections within the provider network 402, which routes traffic from the physically connected client network 450 to the computing and / or storage devices implementing the synthetic sensor service 102 via direct connection 452. For example, as... Figure 8-10 The connected car service 742 (e.g., data lake) described herein can be implemented using cloud services such as computing service 414 and data storage service 416, or can be implemented directly by the client using client resources such as client computing device 456, client storage device 458 or other client devices 460.
[0070] Figure 5 An example deployment of a synthetic sensor package according to some embodiments is shown. Figure 5 The text describes a specific example of deploying a synthetic sensor package titled "Children in a vehicle?" to illustrate more generally how the placement of the synthetic sensor package can be determined. However, compared to... Figure 5 The concepts described herein are similar to those applicable to various other types of synthetic sensors deployed.
[0071] For example, the "Child in vehicle?" synthetic sensor package 502 can be sent via network 122 to the onboard computing device 152. The "Child in vehicle?" synthetic sensor package 502 can be included in, for example... Figure 2 The code logic and mappings within the envelope described herein may further include annotations such as required inputs, optional inputs, certification levels, and fault contingency plans. For example, the annotations for the "Child in Vehicle?" synthetic sensor package 502 may include the required inputs for the seat weight sensor and the cockpit camera. Optional inputs may include a door closing speed sensor. Additionally, the certification level may indicate that if only the required inputs are available, the Quality Management (QM) certification level will be assigned to the "Child in Vehicle?" synthetic sensor; while if the optional input of the door closing speed sensor is available, the ASIL A certification level will be assigned to the "Child in Vehicle?" synthetic sensor. Furthermore, the annotations may indicate that in the event of the loss of optional inputs, the certification level for the "Child in Vehicle?" synthetic sensor will be downgraded from ASIL A to QM.
[0072] In some embodiments, the synthetic sensor orchestration environment 140 can read a note associated with the “Child in vehicle?” synthetic sensor package 502 and determine, based on the note, that the “Child in vehicle?” synthetic sensor package 502 will be placed in the safety operating system / domain 504, which has accessed input data from the seat weight sensor 508 and the cockpit camera 506. Additionally, the synthetic sensor orchestration environment 140 can further recognize that the “Child in vehicle?” synthetic sensor package 502 has an optional input for a door closing speed sensor, and also recognizes that the door closing speed synthetic sensor 512 is implemented in the synthetic sensor orchestration environment 140 associated with the safety operating system / domain 504. Therefore, the synthetic sensor orchestration environment can determine that the “Child in vehicle?” synthetic sensor 510 is placed in the safety operating system / domain 504 as an ASIL A sensor. Furthermore, the synthetic sensor orchestration environment 140 can further inform other synthetic sensors that the new ASIL A type “Child in vehicle?” synthetic sensor 510 has been implemented in the safety operating system / domain 504.
[0073] In some embodiments, if multiple operating systems / domains have access to the seat weight sensor 508 and the cockpit camera 506, placement decisions may be based on whether the operating systems / domains have access to the door closing speed synthetic sensor 512. Furthermore, where multiple operating systems / domains have access to both the required and optional inputs, placement decisions may be made based on the certification level of the hardware and algorithms associated with the respective domain. For example, to meet ASIL A certification, a certain type of hardware and / or annotation-class code may be required. Therefore, even if both the required and optional inputs are available in multiple domains, placement decisions can be made based on the certification level of the hardware and algorithms associated with a particular operating system / domain, allowing the synthetic sensor to be placed to achieve a higher certification level. In some embodiments, placement decisions may be based on whether placing the synthetic sensor in a particular operating system / domain enables an existing synthetic sensor to achieve a higher certification level. For example, if the synthetic sensor package being placed is a door closing speed synthetic sensor package, and the "Child in vehicle?" synthetic sensor 510 is already in place, then the placement decision for the door closing speed synthetic sensor 512 might be based on the fact that placement in the safety system operating system / domain 504 would enable the "Child in vehicle?" synthetic sensor 510 to be upgraded to a higher certification level. It should be noted that these placement decisions can be made based on both the annotations associated with the implemented synthetic sensor and the annotations associated with the synthetic sensor package being placed.
[0074] Furthermore, in some embodiments, the synthetic sensor package may specifically indicate a particular placement location for the synthetic sensor. For example, a vehicle OEM may include configuration instructions in the synthetic sensor package that instruct the synthetic sensor orchestration environment where to place the synthetic sensor.
[0075] Figure 6 A sample client console for a synthetic sensor service, according to some embodiments, is shown.
[0076] exist Figure 6 In the settings, the "Sensor Access" tab has been selected, indicating existing physical sensors in "OEM A" vehicles with a 2009, 2010, or 2011 model year, a specific operating system platform, and a trim level. Furthermore, as... Figure 6As shown, some sensor data from some sensors is available to first-party applications (e.g., OEM manufacturer applications), while sensor data from other sensors is available to second-party applications (e.g., OEM part manufacturer applications), and sensor data from another set of sensors can be used by automotive open system architectures (e.g., AutoSAR), third parties (e.g., application developers who are not first-party OEMs or second-party OEM part manufacturers). For example, third-party applications may come from the OEM's developer ecosystem or from an existing developer application store ecosystem. And, although... Figure 6 The system is not shown, but you can select the System Access tab to see which systems (e.g., security system domain, entertainment domain, vehicle control domain, etc.) are accessible to first-party, second-party, third-party, and / or Auto SAR applications. Additionally, you can select the Computing Unit Control tab to determine whether first-party, second-party, third-party, and / or Auto SAR applications can interact with the computer controller in the corresponding domain. Figure 6 The interface shown can be used to select input sensors by dragging and dropping them, such as... Figure 2 The synthetic sensor definition shown is illustrated in the diagram.
[0077] Figure 7 An example view of a synthetic sensor orchestration environment according to some embodiments is shown.
[0078] As discussed above, a synthetic sensor service, such as synthetic sensor service 102, may include a service interface 108 implementing a console, such as an OEM / third-party sensor control console 738. This allows developers of rules, algorithms, models, and synthetic sensors (740) to deploy such rules, algorithms, models, and / or synthetic sensors into an in-vehicle synthetic sensor orchestration environment 744. Additionally, vehicle usage data, such as telemetry data, may be provided back to the developers of the rule algorithms, models, and synthetic sensors (740) and / or the connected vehicle service 742. For example, in some embodiments, applications provided by the developers of the rule algorithms, models, and synthetic sensors (740) may be purchased from an application store 732, which could result in the deployment of a third-party application 734 or a first / second-party application 736 in the vehicle. Furthermore, purchasing from an application store can enable the deployment of one or more synthetic sensors into the in-vehicle synthetic sensor orchestration environment 744, where the third-party application 734 or the first / second-party application 735 uses the output of the synthetic sensors as input to the respective application. For example, synthetic sensor X (720), synthetic sensor Y (722), and synthetic sensor Z (724) can be deployed to an in-vehicle synthetic sensor orchestration environment 744. Additionally, in some embodiments, rule A (712), code B (714), model A (716), and model B (718) can be deployed to the in-vehicle synthetic sensor orchestration environment 744. In some embodiments, the synthetic sensor package may include code for synthetic sensor X (720), synthetic sensor Y (722), and synthetic sensor Z (724), as well as mappings for input sensor control mapping 708 and output sensor control mapping 710, and / or may include code for rule A (712), code B (714), model A (716), and model B (718).
[0079] In some embodiments, a third-party application may not have the authority to deploy synthetic sensors, and may instead request authorization from a first or second party, such as a vehicle OEM, to deploy the synthetic sensors required for its implementation. In some embodiments, when implemented in a vehicle, the application may use an input discovery module (e.g., input discovery component 904) to request access to the inputs required by the application, which... Figure 9 Further, this is desirable. In such embodiments, if the synthetic sensor is not already implemented in the vehicle's synthetic sensor orchestration environment, the input discovery module can request additional synthetic sensors required by the application from the OEM. In some embodiments, the OEM may select a specific type of synthetic sensor to implement in response to the request, which may differ from the specific type of synthetic sensor requested. For example, based on the vehicle's overall layout or OEM preferences, the OEM may select different synthetic sensors for use by third-party applications.
[0080] In some embodiments, the OEM client of the synthetic sensor service can control access to data from the synthetic sensor. This may apply to synthetic sensors already deployed in a vehicle's synthetic sensor orchestration environment and / or new synthetic sensors requested for deployment in the vehicle. For example, even if synthetic sensor Y (722) is deployed in an on-board synthetic sensor orchestration environment 744, a third-party application 734 may not be able to access the output of synthetic sensor Y (722), while a first and second-party application 736 may be able to access the output of synthetic sensor Y (722).
[0081] In some embodiments, the input sensor control map 708 can map sensor data from sensors 1, 2, and 3 in the security system domain 704 to rule A (712), code B (714), and model A (716). Furthermore, the input sensor control map 708 can map sensor data from sensors 4 and 5 in the entertainment system domain 706 to model A (716) and model B (718). However, in some cases, sensor data may be blocked from transmission due to mapping, code logic, or other reasons (e.g., failure to pay for the application). Therefore, Figure 7 It is shown that sensor data from sensor 5 is not allowed to be transmitted to model B (718) and the output of synthetic sensor Y (722) is not allowed to be transmitted to application 734.
[0082] Furthermore, in some embodiments, the synthetic sensor may be a guiding synthetic sensor that allows or disallows sensor data transmission to a specific destination. For example, guiding sensor 728 may allow or disallow sensor data from sensor 1 to be transmitted to the Auto SAR adaptive application 730.
[0083] Figure 8 An example view of a specific synthetic sensor is shown, implemented according to some embodiments, in a synthetic sensor orchestration environment running on top of an in-vehicle operating system.
[0084] like Figure 8As shown, in some embodiments, the in-vehicle synthetic sensor orchestration environment 744 may be implemented on top of an in-vehicle operating system 702 including a hardware abstraction layer (HAL), an in-vehicle bus 802, or both. The HAL or in-vehicle bus 802 may interface with a local resource access code component 804 of the in-vehicle synthetic sensor orchestration environment 744, and the local resource access code component 804 may allow or disallow sensor data from sensors 1, 2, 3, 4, or 5 to be transmitted from the HAL or in-vehicle bus 802 to the corresponding components of the synthetic sensor, such as synthetic sensor code 806 and synthetic sensor machine learning model 808. Additionally, messaging topics 810 may allow or disallow data transmission from the synthetic sensor orchestration environment 744 to other in-vehicle applications 814, such as third-party applications 734, first / second-party applications 736, and / or AutoSAR adaptive applications 730. In some embodiments, the in-vehicle synthetic sensor orchestration environment 744 may further include a cloud mirroring / streaming manager component 812, which mirrors or transmits sensor data (e.g., telemetry data) to a connected car service 742. In some embodiments, the connected car service may be located within a network of providers implementing synthetic sensor services (e.g., Figure 4 The provider network 402 shown is implemented in and / or may be implemented in the client network (e.g., located at the OEM data center, for example) Figure 4 Implemented in the client network 450 shown.
[0085] In some embodiments, sensor data may be provided to a storage service, such as data storage service 416 (e.g. Figure 4 As shown in the diagram), the sensor data can then be used by machine learning services, such as machine learning service 418 (as shown in the diagram). Figure 4 (As shown in the diagram). The machine learning service can determine relationships that can be further used to create new logical elements, which can be added to the code logic repository 104. The new logical elements can then be selected for use in the new synthetic sensor.
[0086] Figure 9 An example view of a specific synthetic sensor is shown, implemented according to some embodiments, in a synthetic sensor orchestration environment running on top of a vehicle's entertainment operating system.
[0087] Figure 9 The vehicle-mounted synthetic sensor orchestration environment 744 shown can be similar to Figure 8 The vehicle-mounted synthetic sensor orchestration environment 744 shown differs in that... Figure 9In this context, the in-vehicle synthetic sensor orchestration environment 744 is implemented on top of the infotainment system operating system 902, and uses the input discovery component 904 to transmit sensor data to third-party applications 734 and / or first / second-party applications 736, which are implemented in a domain different from the infotainment operating system domain 902.
[0088] Figure 10 An example view of a specific synthetic sensor is shown, according to some embodiments, in a synthetic sensor orchestration environment running in a gateway between the vehicle's operating system and the vehicle's infotainment operating system.
[0089] Figure 10 The vehicle-mounted synthetic sensor orchestration environment 744 shown can be similar to Figure 8 Or the vehicle-mounted synthetic sensor orchestration environment 744 shown in Figure 9, the difference being in Figure 10 In this embodiment, the in-vehicle synthetic sensor orchestration environment 744 is implemented on top of the in-vehicle operating system 702 implemented in the gateway 1044, and uses the input discovery component 904 to transmit sensor data to third-party applications 734 and / or first / second-party applications 736, which are implemented in the entertainment operating system 902 implemented on the in-vehicle infotainment computing device 1002. In some embodiments, the gateway 1044 and the in-vehicle infotainment 1002 may be independent computing devices that respectively implement the in-vehicle operating system 702 and the entertainment system operating system 902.
[0090] Figure 11 A flowchart illustrating the operation of a synthetic sensor service according to some embodiments is shown.
[0091] At 1102, the synthetic sensor service provides an interface to vehicle manufacturers, parts manufacturers, or other third parties. This interface enables clients of the synthetic sensor service to design new types of synthetic sensors. The interface can also be used to deploy specific types of synthetic sensors (e.g., those designed by the client) to specific vehicles. In some embodiments, the interface may provide an API to an application store to automatically deploy synthetic sensor packages to specific vehicles in response to applications purchased for those vehicles using synthetic sensors.
[0092] At 1104, the synthetic sensor service receives via an interface the selection (or mapping, in order to create synthetic sensors) of one or more synthetic sensors deployed in the synthetic sensing orchestration environment of the vehicle.
[0093] At 1106, the Synthetic Sensor Service generates one or more synthetic sensor packages of one or more selected (or created) synthetic sensors for deployment to a synthetic sensor orchestration environment for vehicles, wherein the one or more synthetic sensor packages are formatted with encapsulated code logic and mapping and have one or more annotations of the synthetic sensors outside the envelope.
[0094] At 1108, the synthetic sensor service provides one or more synthetic sensor packages for deployment to the vehicle during the vehicle manufacturing process. In some embodiments, one or more synthetic sensors associated with one or more synthetic sensor packages may assist in the manufacturing and / or testing of the vehicle under construction. In some embodiments, one or more synthetic sensor packages may be deployed during the manufacturing process after the vehicle is powered on (e.g., with a battery installed), after an onboard computer is installed in the vehicle, and after the vehicle has wiring harnesses attached to the onboard computer. In some embodiments, one or more sensors associated with one or more synthetic sensor packages may run a quality assurance model to check the quality of the installed physical sensors and computing systems. In some embodiments, the synthetic sensors may be released from the synthetic sensor orchestration environment when the vehicle passes quality assurance testing.
[0095] At 1110, the synthetic sensor service provides one or more synthetic sensor packages for deployment to the vehicle via a network connection to the vehicle, wherein one or more sensor packages are provided to add synthetic sensors or other features to the vehicle after the vehicle is available for use by the vehicle operator (e.g., after the vehicle has been sold to a consumer).
[0096] In some embodiments, the synthetic sensor service may perform both 1108 and 1110 for a vehicle, or may perform only one without the other.
[0097] Figure 12 A flowchart illustrating the operation of an onboard synthetic sensor orchestration environment according to some embodiments is shown.
[0098] At 1202, the vehicle's onboard computer implements a synthetic sensor orchestration environment configured to receive synthetic sensor packets via a network connection to the vehicle, and configured to implement new synthetic sensors after the vehicle is put into use by the vehicle's owner or operator or during the vehicle's manufacturing process. In some embodiments, the synthetic sensor orchestration environment may be implemented on its own independent computing device, or on a computing device that also implements an onboard operating system, a computing device that implements an infotainment system, or a gateway computing device that provides a gateway between the vehicle's operating systems / domains, such as a gateway between the onboard operating system computing device and the infotainment computing device.
[0099] At 1204, the synthetic sensor orchestration environment establishes a communication channel between the vehicle's existing physical sensors and the synthetic sensor orchestration environment via the vehicle's onboard bus.
[0100] At 1206, the synthetic sensor orchestration environment receives one or more synthetic sensor packets at the vehicle via a network connection to the synthetic sensor service or via a local production network at the manufacturing site, wherein the synthetic sensor packet includes encapsulated code logic and mapping, as well as one or more annotations of the synthetic sensor outside the envelope.
[0101] At 1208, the synthetic sensor orchestration environment determines the specific placement of a given synthetic sensor based on one or more annotations for that synthetic sensor, wherein the synthetic sensor orchestration environment spans multiple operating systems or operating system domains. The synthetic sensor orchestration environment can repeat this determination for each of the synthetic sensor packages being deployed.
[0102] At 1210, the synthetic sensor orchestration environment determines the communication mapping for each given synthetic sensor within the corresponding operating system or operating system domain for placing the synthetic sensor, based on the mapping contained in the synthetic sensor package.
[0103] At 1212, the synthetic sensor orchestration environment performs one or more configuration operations based on the determined communication mapping to enable a responder in a given synthetic sensor to access input data from one or more existing physical sensors available in a specific operating system or operating system domain, or to access input data from another synthetic sensor implemented in the synthetic sensor orchestration environment. The synthetic sensor orchestration environment also performs one or more configuration operations to enable output data from a responder in a given synthetic sensor to be transmitted to an output destination in the vehicle domain and to a service interface residing in the cloud or in a consumer data center (e.g., Figure 1 (Service interface 108 in the middle), such as other synthetic sensor applications, etc.
[0104] At point 1214, the synthetic sensor orchestration environment can optionally publish or inform one or more new synthetic sensors to other synthetic sensors within the synthetic sensor orchestration environment and / or to applications configured to interact with the synthetic sensor orchestration environment. Furthermore, the synthetic sensor orchestration environment can inform the application of one or more new synthetic sensors. For example, the synthetic sensor orchestration environment may include, for example, […]. Figure 9 The input detection component 904 described herein is used to alert the application to the presence of one or more new synthetic sensors.
[0105] Figure 13A method for reconfiguring synthetic sensor placement in two or more operating systems or operating system domains in response to changes in the vehicle's operating system or operating system domain, according to some embodiments, is illustrated.
[0106] At 1302, the synthetic sensor orchestration environment detects whether additional physical sensors have been added to the vehicle and whether they are connected to the vehicle's communication bus.
[0107] At 1304, the synthetic sensor orchestration environment detects whether communication from existing physical sensors via the vehicle's communication bus has been lost.
[0108] At 1306, the synthetic sensor orchestration environment receives notifications or announcements of additional synthetic sensors that have been deployed to one or more domains of the synthetic sensor orchestration environment.
[0109] At 1308, the synthetic sensor orchestration environment determines whether one or more changes have occurred to the certification level for synthetic sensors implemented in the synthetic sensor orchestration environment or for physical sensors that share sensor data with the synthetic sensor orchestration environment.
[0110] At 1310, the synthetic sensor orchestration environment determines / detects changes in vehicle resource availability, such as changes in computing power in one or more domains of the vehicle. For example, a change in additional computing power in a particular domain may justify repositioning the synthetic sensor to a domain with additional computing power, or this placement change may be justified in conjunction with other considerations.
[0111] At point 1312, the synthetic sensor orchestration environment determines whether the number or significance of changes occurring at points 1302, 1304, 1306, 1308, or 1310 exceeds one or more thresholds, thus triggering a reconfiguration operation. During the reconfiguration operation, the synthetic sensor orchestration environment may re-evaluate the placement of implemented synthetic sensors to determine whether placing one or more implemented synthetic sensors in different domains would improve system performance, enhance the certification level of the synthetic sensors, or otherwise provide better performance. If one or more thresholds are not exceeded at point 1312, the synthetic sensor orchestration environment continues to monitor for additional changes.
[0112] If one or more thresholds are exceeded at 1312, then between 1314 and 1318, the synthetic sensor orchestration environment determines the impact of repositioning one or more implemented synthetic sensors to a different domain. For example, repositioning a synthetic sensor may provide optional inputs to the repositioned synthetic sensor or to synthetic sensors in the domain where the synthetic sensor was implemented. This additional optional input may allow at least some synthetic sensors to be upgraded to a higher certification level.
[0113] At 1320, the synthetic sensor orchestration environment selects the appropriate placement of the synthetic sensor to be repositioned as part of the reconfiguration based on the determined influence.
[0114] Example computer system
[0115] Any of various computer systems can be configured to implement processes associated with synthetic sensor services, synthetic sensor orchestration environments, networks of providers implementing synthetic sensor services, operating systems in vehicles or devices, or any other components shown in the diagram above. For example, Figure 14 This is a block diagram illustrating an example computer system implementing some or all of the techniques described herein according to some embodiments. In various embodiments, synthetic sensor services, synthetic sensor orchestration environments, provider networks implementing synthetic sensor services and other cloud services, operating systems in vehicles or devices, or the aforementioned... Figure 1-13 Any other component may each contain one or more computer systems 1400, for example Figure 14 The computer system described in [the document].
[0116] In the illustrated embodiment, computer system 1400 includes one or more processors 1410 coupled to system memory 1420 via input / output (I / O) interface 1430. Computer system 1400 further includes a network interface 1440 coupled to I / O interface 1430. In some embodiments, computer system 1400 may illustrate a server implementing enterprise logic or downloadable applications, while in other embodiments, the server may contain more, fewer, or different elements than computer system 1400.
[0117] In various embodiments, computing device 1400 may be a single-processor system including one processor, or a multiprocessor system including a plurality of processors 1410A-1410N (e.g., two, four, eight, or another suitable number of processors). Processors 1410A-1410N may include any suitable processor capable of executing instructions. For example, in various embodiments, processors 1410A-1410N may be processors implementing any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, or MIPS ISA or any other suitable ISA. In some embodiments, processors 1410A-1410N may include dedicated processors, such as graphics processing units (GPUs), application-specific integrated circuits (ASICs), etc. In a multiprocessor system, each of processors 1410A-1410N may collectively (but not necessarily) implement the same ISA.
[0118] System memory 1420 may be configured to store program instructions and data accessible by processors 1410A-1410N. In various embodiments, system memory 1420 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data implementing one or more of the desired functions as described above are shown stored in system memory 1420 as code (i.e., program instructions) 1425 and data 1426.
[0119] In one embodiment, I / O interface 1430 may be configured to coordinate I / O operations between processors 1410A-1410N, system memory 1420, and any peripheral devices (including network interface 1440 or other peripheral interfaces) within the device. In some embodiments, I / O interface 1430 may perform any required protocols, timing, or other data transformations to convert data signals from one component (e.g., system memory 1420) into a format suitable for another component (e.g., processor 1410). In some embodiments, I / O interface 1430 may include support for devices attached via various types of peripheral buses, such as variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, I / O interface 1430 may include support for devices attached via an automotive CAN bus, etc. In some embodiments, the functionality of I / O interface 1430 may be divided into two or more independent components (e.g., a northbridge and a southbridge). Furthermore, in some embodiments, some or all of the functionality of the I / O interface 1430 (such as an interface to system memory 1420) may be directly incorporated into the processors 1410A-1410N.
[0120] Network interface 1440 can be configured to allow data exchange between computing device 1400 and other devices 1460 attached to one or more networks 1450. In various embodiments, network interface 1440 can support communication via any suitable wired or wireless general-purpose data network, such as Ethernet, cellular, Bluetooth, WiFi, or ultra-wideband networks. Additionally, network interface 1440 can support communication via telecommunications / telephone networks, such as analog voice networks or digital fiber optic communication networks, via storage area networks (SANs), such as Fibre Channel, or via any other suitable type of network and / or protocol.
[0121] In some embodiments, system memory 1420 may be one embodiment of a computer-readable (i.e., computer-accessible) medium configured to store program instructions and data as described above for implementing embodiments of corresponding methods, systems, and devices. However, in other embodiments, program instructions and / or data may be received, transmitted, or stored on different types of computer-readable media. Generally, computer-readable media may include non-transitory storage media or memory media such as magnetic or optical media, for example, a disk or DVD / CD coupled to computing device 1400 via I / O interface 1430. One or more non-transitory computer-readable storage media may also include any volatile or non-volatile media that may be included as system memory 1420 or another type of memory in some embodiments of computing device 1400, such as RAM (e.g., SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc. Additionally, computer-readable media may include transmission media or signals such as electrical, electromagnetic, or digital signals transmitted via communication media such as networks and / or wireless links, as may be implemented via network interface 1440. Figure 14 Some or all of the computing devices, such as the computing device shown, can be used to implement the described functionality in various embodiments; for example, software components running on various different devices and servers can cooperate to provide the functionality. In some embodiments, portions of the described functionality can be implemented using storage devices, network devices, or various types of computer systems. As used herein, the term "computing device" refers to at least all of these types of devices, but is not limited to them.
[0122] Embodiments of this disclosure may be described in accordance with the following terms: Clause 1. A system comprising: One or more computing devices storing program instructions for implementing synthetic sensor services for a vehicle, wherein the program instructions, when executed by the one or more computing devices, cause the one or more computing devices to: Implement an interface to receive instructions for one or more synthetic sensors to be deployed to a vehicle; The interface receives instructions for one or more synthetic sensors to be deployed to the vehicle, wherein the interface instructs, for the one or more synthetic sensors, to map the one or more synthetic sensors to one or more existing physical sensors or other synthetic sensors of the vehicle; and Based on received instructions, one or more synthetic sensor packages are provided for deployment to a synthetic sensor orchestration environment of the vehicle, wherein the provided one or more synthetic sensor packages include code defining the one or more synthetic sensors and one or more corresponding annotations defining the inputs, outputs or attributes of the corresponding one or more synthetic sensors, wherein the code is included in an encapsulated format together with the one or more annotations outside the scope of the code envelope.
[0123] Clause 2. The system according to Clause 1, wherein the synthetic sensor service is configured to provide the one or more synthetic sensor packages to the vehicle when: After the vehicle becomes available for use by the owner or operator of the vehicle; or During the manufacturing process of the vehicle, after one or more computing devices storing program instructions for implementing the synthetic sensor orchestration environment are installed in the vehicle, the one or more synthetic sensors are configured to generate outputs for testing the vehicle during the manufacturing process.
[0124] Clause 3. The system described in Clause 1 or Clause 2 further includes: One or more computing devices, located in the vehicle, store program instructions for implementing at least a portion of the synthetic sensor orchestration environment, wherein the synthetic sensor orchestration environment is configured to: Receive one or more synthetic sensor packages, each including one or more of the corresponding annotations; The specific placement location of the one or more synthetic sensors is determined based on the one or more corresponding annotations; and For each of the one or more synthetic sensors: Assign the corresponding synthetic sensors to the determined placement locations; Perform one or more configuration operations to enable the synthetic sensor to access input data from the one or more existing physical sensors or other synthetic sensors of the vehicle according to the mapping; and Perform one or more configuration operations to provide output data from the synthetic sensor to one or more output destinations within the vehicle's operating system environment or to one or more other destinations outside the vehicle's operating system environment.
[0125] Clause 4. The system according to any one of Clauses 1 to 3 further includes a computing device located in a plurality of additional vehicles, which stores at least a portion of program instructions for implementing an additional synthetic sensor orchestration environment in the plurality of additional vehicles. The plurality of additional vehicles comprise vehicles manufactured by two or more different vehicle manufacturers or different vehicle platforms manufactured by the same vehicle manufacturer, and The vehicle's synthetic sensor service provides a standard interface for selecting a synthetic sensor to be deployed to any of the plurality of additional vehicles manufactured by the two or more different vehicle manufacturers, wherein existing physical sensors in the plurality of additional vehicles manufactured by the two or more different manufacturers have different interface formats for communicating with the existing physical sensors.
[0126] Clause 5. The system according to any one of Clauses 1 to 4, wherein said interface is configured to: Receive instructions from the vehicle manufacturer for a synthetic sensor, the synthetic sensor including logic for deploying additional features of the vehicle to be added during the vehicle's manufacturing process or after the vehicle is available for use by the operator of the vehicle; or Instructions are received from a third party for synthesizing sensors, which include logic for additional features of the vehicle to be deployed during the vehicle's manufacturing process or after the vehicle is made available to the operator of the vehicle.
[0127] Clause 6. A system comprising: One or more computing devices configured to be installed in a vehicle, wherein the one or more computing devices store program instructions for implementing a synthetic sensor orchestration environment in the vehicle, wherein the synthetic sensor orchestration environment is configured to: Receive a synthetic sensor packet from the synthetic sensor service, the synthetic sensor packet comprising: One or more logical elements and mappings for synthesizing sensors; and Define one or more annotations for the input, output, or properties of the synthetic sensor. The one or more logical elements and mappings thereon are included in an encapsulated format along with the one or more annotations that are outside the envelope of the synthetic sensor package; The specific placement location of the synthetic sensor is determined based on the one or more annotations; and Perform one or more configuration operations to enable the synthetic sensor to access input data from one or more of the following: Physical sensors; or Another synthetic sensor for the vehicle.
[0128] Clause 7. In the system described in Clause 6, the one or more annotations of the synthetic sensor package include one or more of the following: The indication of the certification level of the synthetic sensor; Indication of one or more required inputs from the synthetic sensor; Indications from one or more additional inputs of the synthetic sensor; An indication of the output type of the synthetic sensor; or The quality level or confidence level indication of the output of the synthetic sensor.
[0129] Clause 8. In the system pursuant to Clause 7, the one or more annotations of said synthetic sensor package further include one or more of the following: A dependency chain, which indicates other synthetic sensors from which the synthetic sensor will receive one or more inputs; or The emergency response plan includes the following instructions: One or more alternative inputs are provided for use by the synthetic sensor when the main input is unavailable; or A downgraded certification level for use by the synthetic sensor when a certain number or type of main input is unavailable.
[0130] Clause 9. The system according to any one of Clauses 6 to 8, wherein the synthetic sensor orchestration environment is further configured to: The synthetic sensor is automatically repositioned to another location in response to changes in the sensor orchestration environment.
[0131] Clause 10. In the system described in Clause 9, the changes in the sensor orchestration environment include one or more of the following: The addition of another synthetic sensor; The change in the availability of additional inputs to the synthetic sensor; Modification of the certification level of the synthetic sensor; or The change in resource availability at the placement location or the change in resource availability at the alternative placement location.
[0132] Clause 11. The system according to Clause 9 or Clause 10, wherein said placement location includes one or more of the following: Information and entertainment domain; Vehicle safety system domain; Gateway domain between one or more other domains of the vehicle; or The cloud domain outside the range of the vehicle.
[0133] Clause 12. In the system according to Clause 11, wherein, in order to determine the placement location for implementing the synthetic sensor or to automatically reposition the synthetic sensor to a different placement location, the synthetic sensor orchestration environment is configured to: If the synthetic sensor is to be placed in the first placement location, the impact on the input or output availability of the synthetic sensor or other synthetic sensors in the first placement location is determined, wherein the impact is determined based on one or more annotations of the synthetic sensor or based on one or more annotations of other synthetic sensors in the first placement location; If the synthetic sensor is to be placed in the second placement location, the impact on the input or output availability of the synthetic sensor or other synthetic sensors in the second placement location is determined, wherein the impact is determined based on one or more annotations of the synthetic sensor or based on one or more annotations of other synthetic sensors in the second placement location; and The placement of the synthetic sensor in either the first or the second placement position is selected based on the determined influence of the first and second placement positions.
[0134] Clause 13. The system according to any one of Clauses 6 to 12, wherein the synthetic sensor orchestration environment is configured to: Allowing or disallowing access to sensor data from the vehicle's physical sensors without modifying either the physical sensors or the logic for performing authenticated functions using data from the physical sensors outside the scope of the synthetic sensor orchestration environment, so that the authentication level of the physical sensors and the authentication level of the logic outside the scope of the synthetic sensor orchestration environment are not downgraded by adding or removing synthetic sensors in the synthetic sensor orchestration environment.
[0135] Clause 14. The system according to any one of Clauses 6 to 13, wherein the synthetic sensor orchestration environment is configured to: Receive a guidance synthesis sensor package including code for guidance sensors configured to allow or disallow the transmission of sensor data from physical sensors in the vehicle; and Perform one or more configuration operations to enable the guidance synthesis sensor to access input data from the physical sensors of the vehicle, and perform one or more configuration operations to enable or disable sensor data from the physical sensors to be transmitted through the guidance synthesis sensor to one or more specific destinations.
[0136] Clause 15. The system according to any one of Clauses 6 to 14, wherein the synthetic sensor orchestration environment is configured to: Receive another synthetic sensor package including one or more logical elements and mappings for another synthetic sensor and one or more annotations for said other synthetic sensor, wherein the one or more logical elements and the mappings are included in an encapsulated format together with the one or more annotations outside the scope of the envelope of said other synthetic sensor package, wherein the one or more annotations indicate that the output of said synthetic sensor is the input of said other synthetic sensor; The specific placement location for implementing the other synthetic sensor is determined based on the one or more annotations; and Perform one or more configuration operations to enable the synthetic sensor to access input data from the other synthetic sensor and one or more other synthetic or physical sensors of the vehicle.
[0137] Clause 16. The system according to any one of Clauses 6 to 15, wherein the synthetic sensor orchestration environment is configured to: Receive another synthetic sensor package including one or more logical elements and mappings for another synthetic sensor and one or more annotations for said other synthetic sensor, wherein said one or more logical elements and said mappings are included in an encapsulated format together with said one or more annotations outside the scope of said other synthetic sensor's envelope; The specific placement location for implementing the other synthetic sensor is determined based on the one or more annotations used for the other synthetic sensor; and Perform one or more configuration operations to enable the other synthetic sensor to access input data from one or more other synthetic or physical sensors of the vehicle. The annotations of the synthetic sensor indicate that the other synthetic sensor is an additional input for the synthetic sensor, and The synthetic sensor orchestration environment is further configured to: One or more configuration operations are automatically performed in response to the implementation of the other synthetic sensor, enabling the synthetic sensor to access the output of the other synthetic sensor.
[0138] Clause 17. The system according to Clause 16, wherein: The one or more annotations in the synthetic sensor package indicate that the synthetic sensor is eligible for different certification levels when the additional input of the synthetic sensor is available; and The synthetic sensor orchestration environment is configured to automatically update the authentication level associated with the synthetic sensor after performing one or more configuration operations to enable the synthetic sensor to access the output of the other synthetic sensors.
[0139] Clause 18. The system according to any one of Clauses 6 to 17, wherein the synthetic sensor orchestration environment is configured to: Publish or inform other destinations in the synthetic sensor orchestration environment of one or more characteristics of the synthetic sensor indicated in the one or more annotations.
[0140] Clause 19. The system according to any one of Clauses 6 to 18, wherein the synthetic sensor orchestration environment is further configured to: The output from the synthetic sensor is provided to a vehicle system outside the range of the synthetic sensor orchestration environment, wherein the output causes the vehicle system to take one or more control actions based on the output.
[0141] Clause 20. The system according to any one of Clauses 6 to 19, wherein the synthetic sensor orchestration environment is further configured to: The output from the synthetic sensor is provided to a remote system outside the range of the vehicle.
[0142] Clause 21. A non-transitory computer-readable storage medium containing one or more stored program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to: Receive a synthetic sensor package including one or more logical elements and mappings for a synthetic sensor and one or more annotations defining the inputs, outputs, or properties of the synthetic sensor, wherein the one or more logical elements and mappings are included in an encapsulated format along with the one or more annotations outside the envelope of the synthetic sensor; The specific placement location for implementing the synthetic sensor is determined based on the one or more annotations; and Perform one or more configuration operations to enable the synthetic sensor to access input data from one or more physical or synthetic sensors of the vehicle at the placement location.
[0143] Clause 22. One or more non-transitory computer-readable storage media as described in Clause 21, wherein the mapping of said synthetic sensor includes a mapping of one or more inputs of said synthetic sensor to said one or more logical elements, wherein said one or more logical elements include one or more of the following: Rules to be applied based on the one or more inputs; An algorithm that uses one or more inputs to determine one or more algorithm outputs; or A machine learning model that uses one or more inputs to determine the output of the machine learning model.
[0144] Clause 23. One or more non-transitory computer-readable media as described in Clause 22, wherein the received synthetic sensor mapping indicates a logical mapping of required or additional input to the rule, the algorithm, or the machine learning model, and
[0145] The instructions, when executed on or across the one or more computing devices, further cause the one or more computing devices to: Determine the communication mapping between the synthetic sensor implemented in the specific placement location and the rules, algorithms, or machine learning models included in the specific placement location.
[0146] The various methods illustrated in the accompanying drawings and described herein represent illustrative embodiments of the methods. These methods can be implemented manually in software, in hardware, or a combination thereof. The order of any method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. For example, in one embodiment, the methods can be implemented by a computer system comprising a processor that executes program instructions stored on a computer-readable storage medium coupled to a processor. These program instructions can be configured to implement the functionality described herein (e.g., functionality of data transfer tools, various services, databases, devices, and / or other communication devices, etc.).
[0147] It will be apparent to those skilled in the art who benefit from this disclosure that various modifications and changes can be made. It is intended to cover all such modifications and changes, and therefore the above description is to be considered illustrative rather than restrictive.
[0148] Various embodiments may further include receiving, transmitting, or storing instructions and / or data implemented as described above on a computer-accessible medium. Generally, a computer-accessible medium may include storage media or memory media (such as magnetic or optical media, e.g., magnetic disks or DVD / CD-ROMs), volatile or non-volatile media (such as RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc.), and transmission media or signals (such as electrical, electromagnetic, or digital signals transmitted via communication media (such as networks and / or wireless links).
Claims
1. A system for providing synthetic sensor services, comprising: One or more computing devices storing program instructions for implementing the synthetic sensor service, wherein the program instructions, when executed by the one or more computing devices, cause the one or more computing devices to: Implement an interface to receive instructions for one or more synthetic sensors to be deployed to a vehicle; The interface receives instructions for one or more synthetic sensors to be deployed to the vehicle, wherein the interface instructs the one or more synthetic sensors to be mapped to one or more existing physical sensors or other synthetic sensors of the vehicle. and Based on received instructions, one or more synthetic sensor packages are provided for deployment to a synthetic sensor orchestration environment of the vehicle, wherein the provided one or more synthetic sensor packages include code defining the one or more synthetic sensors and one or more corresponding annotations defining the inputs, outputs or attributes of the corresponding one or more synthetic sensors, wherein the code is included in an encapsulated format together with the one or more annotations outside the scope of the code envelope.
2. The system of claim 1, wherein the synthetic sensor service is configured to provide the one or more synthetic sensor packages to the vehicle in the following situations: After the vehicle becomes available for use by the owner or operator of the vehicle; or During the manufacturing process of the vehicle, after one or more computing devices storing program instructions for implementing the synthetic sensor orchestration environment are installed in the vehicle, the one or more synthetic sensors are configured to generate outputs for testing the vehicle during the manufacturing process.
3. The system according to claim 1, The synthetic sensor service is configured to provide synthetic sensors to multiple additional vehicles, which include vehicles manufactured by two or more different vehicle manufacturers or different vehicle platforms manufactured by the same vehicle manufacturer. The synthetic sensor service provides a standard interface for selecting a synthetic sensor to be deployed to any of the plurality of additional vehicles manufactured by the two or more different vehicle manufacturers, wherein existing physical sensors in the plurality of additional vehicles manufactured by the two or more different manufacturers have different interface formats for communicating with the existing physical sensors.
4. The system of claim 1, wherein the interface is configured to: Receive instructions from the vehicle manufacturer for a synthetic sensor, the synthetic sensor including logic for additional features of the vehicle to be deployed during the vehicle's manufacturing process or after the vehicle is available for use by the vehicle's operator; or Instructions are received from a third party for synthesizing sensors, which include logic for additional features of the vehicle to be deployed during the vehicle's manufacturing process or after the vehicle is made available to the operator of the vehicle.
5. A system for synthesizing sensor orchestration, the system comprising: One or more computing devices configured to be installed in a vehicle, wherein the one or more computing devices store program instructions for implementing a synthetic sensor orchestration environment in the vehicle, wherein the synthetic sensor orchestration environment is configured to: Receive a synthetic sensor packet from the synthetic sensor service, the synthetic sensor packet comprising: One or more logical elements and mappings for synthesizing sensors; and Define one or more annotations for the input, output, or properties of the synthetic sensor. The one or more logical elements and mappings thereon are included in an encapsulated format along with the one or more annotations that are outside the envelope of the synthetic sensor package; The specific placement location of the synthetic sensor is determined based on the one or more annotations; and Perform one or more configuration operations to enable the synthetic sensor to access input data from one or more of the following: Physical sensors; or Another synthetic sensor for the vehicle.
6. The system of claim 5, wherein the one or more annotations of the synthetic sensor package include one or more of the following: The indication of the certification level of the synthetic sensor; Indication of one or more required inputs from the synthetic sensor; Indications from one or more additional inputs of the synthetic sensor; An indication of the output type of the synthetic sensor; or The quality level or confidence level indication of the output of the synthetic sensor.
7. The system of claim 6, wherein the one or more annotations of the synthetic sensor package further include one or more of the following: A dependency chain, which indicates other synthetic sensors from which the synthetic sensor will receive one or more inputs; or The emergency response plan includes the following instructions: One or more alternative inputs are provided for use by the synthetic sensor when the main input is unavailable; or A downgraded certification level for use by the synthetic sensor when a certain number or type of main input is unavailable.
8. The system of claim 5, wherein the synthetic sensor orchestration environment is further configured to: The synthetic sensor is automatically repositioned to another location in response to changes in the sensor orchestration environment.
9. The system of claim 8, wherein the changes in the sensor orchestration environment include one or more of the following: The addition of another synthetic sensor; The change in the availability of additional inputs to the synthetic sensor; Modification of the certification level of the synthetic sensor; or The change in resource availability at the placement location or the change in resource availability at the alternative placement location.
10. The system of claim 8, wherein the placement location includes one or more of the following: Information and entertainment domain; Vehicle safety system domain; Gateway domain between one or more other domains of the vehicle; or The cloud domain outside the range of the vehicle.
11. The system of claim 10, wherein, in order to determine the placement location for implementing the synthetic sensor or to automatically reposition the synthetic sensor to a different placement location, the synthetic sensor orchestration environment is configured to: If the synthetic sensor is to be placed in the first placement location, the impact on the input or output availability of the synthetic sensor or other synthetic sensors in the first placement location is determined, wherein the impact is determined based on one or more annotations of the synthetic sensor or based on one or more annotations of other synthetic sensors in the first placement location; If the synthetic sensor is to be placed in the second placement location, the impact on the input or output availability of the synthetic sensor or other synthetic sensors in the second placement location is determined, wherein the impact is determined based on one or more annotations of the synthetic sensor or based on one or more annotations of other synthetic sensors in the second placement location; and The placement of the synthetic sensor in either the first or the second placement position is selected based on the determined influence of the first and second placement positions.
12. The system of claim 5, wherein the synthetic sensor orchestration environment is configured to: Receive a guidance synthesis sensor package including code for guidance sensors configured to allow or disallow the transmission of sensor data from physical sensors in the vehicle; and Perform one or more configuration operations to enable the guidance synthesis sensor to access input data from the physical sensors of the vehicle, and perform one or more configuration operations to enable or disable sensor data from the physical sensors to be transmitted through the guidance synthesis sensor to one or more specific destinations.
13. The system of claim 5, wherein the synthetic sensor orchestration environment is configured to: Receive another synthetic sensor package including one or more logical elements and mappings for another synthetic sensor and one or more annotations for the other synthetic sensor, wherein the one or more logical elements and the mappings are included in an encapsulated format together with the one or more annotations outside the scope of the envelope of the other synthetic sensor package, wherein the one or more annotations indicate that the output of the synthetic sensor is the input of the other synthetic sensor; The specific placement location for implementing the other synthetic sensor is determined based on the one or more annotations; and Perform one or more configuration operations to enable the synthetic sensor to access input data from the other synthetic sensor and one or more other synthetic or physical sensors of the vehicle.
14. The system of claim 5, wherein the synthetic sensor orchestration environment is configured to: Receive another synthetic sensor package including one or more logical elements and mappings for another synthetic sensor and one or more annotations for said other synthetic sensor, wherein said one or more logical elements and said mappings are included in an encapsulated format together with said one or more annotations outside the scope of said envelope of said other synthetic sensor; The specific placement location for implementing the other synthetic sensor is determined based on the one or more annotations used for the other synthetic sensor; and Perform one or more configuration operations to enable the other synthetic sensor to access input data from one or more other synthetic or physical sensors of the vehicle. The annotations of the synthetic sensor indicate that the other synthetic sensor is an additional input for the synthetic sensor, and The synthetic sensor orchestration environment is further configured to: One or more configuration operations are automatically performed in response to the implementation of the other synthetic sensor, enabling the synthetic sensor to access the output of the other synthetic sensor.
Citation Information
Patent Citations
Method and system for updating the software of a motor vehicle sensor
CN109074252A
Crowdsource-based virtual sensor generation and virtual sensor application control
US20180373266A1
