Method and apparatus for context sensor hub

Abstract

Methods and apparatus for a context sensor hub are disclosed. An example apparatus includes a first processor core to implement a host controller and a second processor core to implement an offload engine. The host controller includes first logic to process sensor data associated with an electronic device when the electronic device is in a low power mode. The host controller offloads compute tasks associated with the sensor data to the offload engine. The offload engine includes a second logic to execute the compute task.

Description

[0001] Divisional Application Instructions

[0002] This application is a divisional application of the Chinese invention patent application with application date of December 31, 2018, application number 201880094818.0, and titled “Method and apparatus for implementing an always-on context sensor hub for processing multiple different types of data inputs”. Technical Field

[0003] The present disclosure relates generally to sensor technology and, more particularly, to methods and apparatus for implementing an always-on contextual sensor hub for processing multiple different types of data inputs. Background Art

[0004] Many processors in modern electronic devices offload tasks associated with the collection and / or processing of sensor data to sensor hubs to reduce power consumption and achieve higher performance efficiency. A sensor hub is a microprocessor, coprocessor, and / or digital signal processor (DSP) specifically designed to handle the processing of sensor data more efficiently than a typical application processor, such as a central processing unit (CPU). With the development of increasingly energy-efficient technologies, some sensor hubs are manufactured to operate in an always-on state, in which the always-on sensor hub remains active even when the associated electronic device is in a low-power (e.g., sleep) mode in order to aggregate and analyze sensor data. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a block diagram illustrating an example hardware architecture of an example context sensor hub constructed according to the teachings disclosed herein.

[0006] Figure 2 This is an overview of the various sensors and / or other hardware drivers and the corresponding data processing logic in Figure 1 Table of example workload division among different cores of an example context sensor hub.

[0007] Figure 3 The diagram shows Figure 1 Block diagram of an example firmware architecture for an example context sensor hub.

[0008] Figure 4 An example sensor tree is shown representing a configuration of example virtual sensor data sources and physical sensor data sources.

[0009] Figures 5 to 7 The diagram shows Figure 3 The example firmware architecture implements various example stages of operation of an always-on facial recognition process.

[0010] Figures 8 to 15 It means that it can be executed to achieve Figure 1 and / or Figure 3 A flowchart of example machine readable instructions for an example context sensor hub.

[0011] The figures are not drawn to scale.In general, the same reference numbers will be used throughout the drawings (one or more) and the accompanying written description to refer to the same or like parts. DETAILED DESCRIPTION

[0012] Sensor hubs are specifically designed to integrate and process sensor data output from one or more sensor devices in an electronic device in an energy-efficient manner. However, many known sensor hubs are capable of performing relatively limited types of data processing based on specific types of sensor data collected from a relatively limited set of sensor devices. For example, some sensor hubs are limited to processing sensor data from basic sensor devices (such as accelerometers, gyroscopes, magnetometers, proximity sensors (infrared (IR) sensors), ambient light sensors, etc.). In such cases, advanced data processing of other types of sensor data can be implemented on other chips dedicated to that purpose. For example, some known dedicated processing chips are specifically designed to process location data obtained from compatible sensor devices (such as Global Navigation Satellite System (GNSS) sensors, Wi-Fi receivers, Bluetooth receivers, cellular network receivers, etc.). Other dedicated processing chips are specifically designed to perform audio processing on audio data obtained from audio sensor devices (such as microphones). Some such audio processing chips implement deep neural networks (DNNs) to enable speech detection (e.g., recognizing spoken words) and / or voice recognition (e.g., identifying a speaker based on their voice). Still other specialized processing chips are specifically designed to process image data obtained from image sensors (e.g., cameras). Some such image processing chips implement convolutional neural networks (CNNs) to enable object detection (e.g., detecting the presence and / or location of an object (e.g., a face) within an image) and / or object recognition (e.g., identifying a detected object and / or distinguishing it from other objects (e.g., identifying a specific person based on their face).

[0013] Different sensor hubs and / or other specialized chips are typically used to process different types of data to improve the performance and power efficiency of processing (one or more) specific types of data. That is, while it may be possible to use a digital signal processor (DSP) specifically designed for audio processing to process image data, the efficiency of this process will be lower than the efficiency of using an image signal processor (ISP) specifically designed for image processing to process image data. It is particularly important to use efficient hardware and / or firmware designs for the sensor hub when the sensor hub is to be implemented in an always-on mode. As used herein, always-on of a sensor hub or other data processing chip is defined as the hub collecting, processing and / or responding to sensor data output by one or more sensors of the electronic device in approximately real time, regardless of whether the electronic device is in sleep mode or idle mode and / or is performing operations unrelated to the collection of sensor data.

[0014] The teachings disclosed herein relate to an embodiment of an always-on context sensor hub. As used herein, the term "context sensor hub" is used to distinguish the example hub disclosed herein from a traditional sensor hub. More specifically, the term "context" is defined herein to refer to a more advanced data processing determination that is related to the context of the associated electronic device and goes beyond the basic sensor data processing determination of a traditional sensor hub. Basic sensor data processing involves motion, orientation, and lighting (e.g., based on accelerometers, gyroscopes, magnetometers, ambient light sensors, etc.). Specifically, the context associated with an electronic device may include the location of the electronic device, the activities and / or behavior of the user of the electronic device, applications running on the electronic device, etc. Therefore, in addition to providing basic sensor data processing for motion detection, orientation detection, lighting detection, etc., the example context sensor hub disclosed herein also enables context processing activities such as position detection, visual analysis (e.g., image processing, object detection / recognition such as facial recognition), audio analysis (e.g., voice detection, sound recognition), and the operation of certain applications installed on the electronic device and / or the implementation of certain platform services based on the above-mentioned context-based determination.

[0015] In addition, as an always-on component, the example context sensor hub disclosed herein is capable of performing both basic sensor data processing and more advanced context-based data processing, as well as the subsequent operation of certain applications, even when the associated electronic device is in sleep mode or idle mode. In order to achieve an always-on implementation that does not consume too much battery power, the data processing of the example disclosed herein will be performed in an efficient and low-power manner. Accordingly, the example always-on context sensor hub disclosed herein is implemented as a system on a chip (SOC) with multiple processor cores that are specifically designed to handle the processing of the different types of sensor data mentioned above. Having separate cores dedicated to different functionalities enables different types of data to be processed in an efficient manner, thereby saving power. In addition, the examples disclosed herein enable different cores in the core to enter sleep when not currently in use and to be started or woken up when the specific functionality provided by these cores is needed. Therefore, although the core is considered to be always-on according to the term defined herein (e.g., it can be active and running when the associated electronic device is in sleep mode or idle mode), not all cores must always be powered. More specifically, a separate core is associated with one of the two power domains. The first power domain is referred to herein as an ultra-low power domain, which is defined herein as running at a power of approximately 1mW or less. The second power domain is referred to herein as a low power domain, which is defined herein as running between approximately 1mW and 20mW. In this context, it means approximately + / - 0.5mW. The ultra-low power domain is, for example, the baseline power disclosed herein. In some examples, at least one core runs in the ultra-low power domain and is always powered (e.g., never power-gated) when the associated electronic device is not completely shut down. In contrast, other cores run in the low power domain and remain asleep or inactive unless they can be woken up when additional computing power is needed. In some examples, these low-power cores are woken up by the ultra-low power core in response to events detected based on sensor data being monitored and / or processed by the ultra-low power core. In some examples, one low-power core can wake up different low-power cores.

[0016] The complexity of separate cores with separate functionalities that can interact with each other, as outlined above, presents numerous difficulties compared to simpler sensor hubs that are dedicated to specific purposes. Among other reasons, distributing functionality across multiple cores makes it difficult to control these separate cores and / or implement efficient interaction between them (e.g., waking up a specific core when needed and / or putting a specific core to sleep when no longer needed). To this end, some example contextual sensor hubs disclosed herein include one core that acts as a host or main controller, handling most of the firmware logic for the various functionalities provided by the hub, while other cores act as offload engines with specific firmware logic and / or hardware to implement the specific tasks or functions designed for these cores.

[0017] Another challenge with multiple cores having function-specific hardware and / or firmware is the complexity this creates for original equipment manufacturers (OEMs) who incorporate the example context sensor hub into suitable electronic devices. Specifically, because each core is designed for a different purpose, each core is associated with a different application programming interface (API), which makes it difficult for OEMs to develop code that can leverage the different functionality provided by the different cores and / or customize such functionality for specific uses. In some examples disclosed herein, these challenges are overcome by including an API proxy in the host controller core so that instructions, commands, configuration parameters, etc. are passed to the other cores without the OEM having to code for the specific APIs of the other cores. Additionally, in some examples, the host controller core includes firmware that provides support for user-friendly programming languages (e.g., Java) to further simplify customization and configuration of the example context sensor hub.

[0018] In some examples, the context sensor hub disclosed herein can be implemented as a companion die (e.g., a coprocessor) to an application processor of an electronic device or as a standalone processor. To achieve standalone functionality, the example context sensor hub includes a runtime environment (e.g., a Java runtime environment) provided within the firmware of one of the cores (e.g., a host controller). The runtime environment enables the execution of any suitable application. Furthermore, in some examples, such applications can be dynamically loaded. Other advantages of the example context sensor hub disclosed herein will become apparent in view of the following detailed description of the accompanying drawings.

[0019] Figure 1is a block diagram of an example hardware architecture of an example context sensor hub 100 constructed according to the teachings disclosed herein. In the illustrated example, the context sensor hub 100 includes five separate processor cores, including a main host microcontroller 102 (referred to herein simply as the host controller 102), a visual capture and detection (VCD) core 104, a vision processor core 106, an audio processor core 108, and an ultra-low power (ULP) processor core 110. Figure 1 As shown in the example of , the ULP processor core 110 operates in an ultra-low power domain 112, while the other cores operate in a low power domain 114. The example context sensor hub 100 also includes a shared memory 116 that operates in the low power domain 114. Additionally, in some examples, one or more input / output (I / O) interfaces 118 are provided in the ultra-low power domain 112, while one or more other I / O interfaces 120 are provided in the low power domain 114. The I / O interfaces 118, 120 enable the context sensor hub 100 to interface with and receive sensor data from sensors or other hardware peripherals of an associated electronic device. Some example sensors include accelerometers, gyroscopes, magnetometers, ambient light sensors, proximity sensors (e.g., IR sensors), microphones, location sensors (e.g., GNSS sensors, Wi-Fi receivers, etc.), image sensors (e.g., cameras), and the like.

[0020] The example ULP processor core 110 provides basic functionality for always-on sensor data monitoring and analysis. In some examples, the ULP processor core 110 is never power-gated, such that the ULP processor core is always powered when power is supplied to the hub 100. That is, the ULP processor core 110 is always powered, regardless of whether the electronic device is in sleep mode or low-power idle mode, as long as the associated electronic device containing the hub 100 is not completely shut down. In some examples, the ULP processor core 110 performs basic processing of the output of one or more sensors to detect events that can trigger the activation of one or more of the other cores 102, 104, 106, 108 in the low-power domain 114 to perform additional data processing and analysis.

[0021] Overall management of the contextual sensor hub 100 is provided by the example host controller 102. Thus, in some examples, when a user (e.g., an OEM) seeks to configure the sensor hub 100, the user interfaces directly with the host controller 102, which can then pass the configuration data to the other cores. Additionally, in some examples, most drivers for the sensors and / or other hardware peripherals monitored by the sensor hub 100 are included on the host controller. However, some sensor drivers can be offloaded to one or more of the other cores. For example, sampling of sensors that are always monitored (e.g., accelerometers, gyroscopes, etc.) can be performed directly by the ULP processor core 110 to maintain such operations in the ultra-low power domain 112, thereby improving efficiency. Other computations that require more power but are not overly intensive or do not involve specialized digital signal processing instructions can be performed directly by the host controller 102. Example computations that can be performed directly on the host controller 102 include location detection (e.g., geolocation detection for identifying GPS coordinates, latitude and longitude, etc.).

[0022] In the illustrated example, the host controller 102 offloads more computationally intensive tasks to one or more of three other cores 104, 106, 108 (collectively referred to herein as offload engines). Figure 1 As shown in FIG, the VCD offload engine 104 includes a VCD digital signal processor (DSP) 122, an I / O interface 124, and a low-power image signal processor (ISP) 126. The I / O interface 124 communicates with (e.g., interfaces with) a camera for capturing images. The ISP 126 facilitates the capture of such images, which can then be processed by the VCD DSP 122. The example VP offload engine 106 includes a VP DSP 128 specifically designed to perform computationally intensive visual analytics, such as object (e.g., facial) recognition. The VP offload engine 106 also includes a convolutional neural network (CNN) accelerator 130 to facilitate object recognition processing. As shown in the illustrated example, there are two separate offload engines or cores associated with visual analytics, including the VCD offload engine 104 and the VP offload engine 106. These two engines are separate because object recognition (performed by the VP offload engine 106) is more computationally intensive than object detection and / or other basic image processing performed by the VCD offload engine 104. By separating functionality into separate cores, performance efficiencies and corresponding power efficiencies can be achieved. However, in some examples, all vision processing can be performed via a single core.

[0023] Finally, if Figure 1As shown in FIG, the audio offload engine 108 includes an audio processing DSP 132 specifically designed to perform computationally intensive audio processing and analysis, such as speech detection and / or sound recognition. The audio offload engine 108 also includes a deep neural network (DNN) accelerator 134 for facilitating speech detection and / or sound recognition processing.

[0024] As mentioned above, most of the drivers for the sensors monitored by the context sensor hub 100 are included in the host controller 102, which serves as the overall manager of the operation of the context sensor hub 100. Furthermore, in some examples, in addition to offloading specific tasks to different cores of the hub 100, most of the data processing logic used to analyze and / or process sensor data also resides in the host controller 102 (e.g., hardware) and / or is executed by the host controller 102 (e.g., software and / or firmware). In some examples, the host controller 102 includes wrappers for the sensor hardware drivers and / or data processing logic of the other cores, such that all drivers and logic are present on and / or can be called from the host controller 102. This significantly simplifies the ability of users (e.g., OEMs) to configure the context sensor hub 100, as users do not need to code separately for each core. Instead, they only need to code for the host controller 102, and can then pass information to other cores and / or call functionality provided by the other cores as needed.

[0025] Figure 2 A table 200 is provided outlining an example workload partitioning of various sensors and / or other hardware drivers and data processing logic among the various cores 102, 104, 106, 108, 110 to achieve always-on functionality that improves power efficiency while meeting a suitable computational load. A first column 202 identifies different types of domains corresponding to different types of sensor data and / or associated functionality provided by the example contextual sensor hub 100. As used herein, the term "sensor data" refers to both the output signals of physical sensors monitored by the hub 100 and the output or results of subsequent analysis and / or subsequent processing of such output signals. Figure 2 As shown in Table 200, the different domains or types of sensor data include a basic sensor data domain 204, a position data domain 206, a visual data domain 208, and an audio data domain 210. An additional platform service domain 212 is also shown. Figure 2 200. Platform services are not sensor data as defined above, but correspond to additional functionality and / or services that may be triggered by sensor data, run in the presence of sensor data, and / or implemented in response to sensor data.

[0026] The second column 214 of the table 200 defines different characteristics associated with the different domains identified in the first column 214. The characteristics associated with the different types of sensor data 204, 206, 208, 210 correspond to either: (1) physical sensors and / or other hardware peripherals that can be used to provide sensor data for processing and analysis by the context sensor hub 100, and (2) sensor data processing logic that defines the processes or operations for analyzing the output of the physical sensors and / or the results produced by the analysis of other sensor data processing logic. Figure 2 The characteristics of the platform services 212 shown in table 200 provide example services that may be implemented by the context sensor hub 100 .

[0027] exist Figure 2 The remaining five columns 216, 218, 220, 222, and 224 in Table 200 correspond to the host controller 102, the ULP processor core 110, the VCD offload engine 104, the VP offload engine 106, and the audio offload engine 108, respectively. Figure 2 The cells within these five columns 216, 218, 220, 222, 224 of the table 200 define the distribution of workload between the different cores 102, 104, 106, 108, 110 associated with the different features (e.g., sensors, sensor logic, and services) identified in the second column 214. For example, for the basic sensor data domain 204, there are two types of sensors, including always-on sensors and other physical sensors. As used herein, "always-on" is defined to mean always powered when power is available (e.g., the associated electronic device is not completely shut down). In contrast, "always-on" is defined herein to mean capable of running when the associated is in low power mode, idle mode, or sleep mode. Therefore, if a component may or may not be in operation when the associated electronic device is in low power mode, then the component may be always-on (capable of running), but not necessarily always-on (always powered). Therefore, in Figure 2 In Table 200 of , an always-on sensor refers to a sensor that is always powered to output sensor data as long as the associated electronic device is powered (regardless of whether the electronic device is in sleep mode or idle mode). In contrast, other physical sensors can be powered on or off as needed when the electronic device is in sleep mode or idle mode, so these physical sensors can be considered to be always on. Example sensors that can be implemented to be always running or at least always on (but not necessarily always running) include accelerometers, gyroscopes, magnetometers, ambient light sensors, proximity sensors (such as IR sensors), and microphones. Figure 2As shown in FIG, drivers for both the always-on sensors and the other sensors are included in and / or executed in the host controller 102. However, as outlined above, because the ULP processor core is the only core running in the ultra-low power domain, sampling of the always-on sensors is offloaded to the ULP processor core 110 to save power. The basic sensor data domain 204 is also associated with two types of sensor data processing logic, including the always-on sensor logic and the other sensor logic. Figure 2 As shown in Table 2 of FIGURE 2, other sensor logic executes directly on the host controller 102. In contrast, the always-on sensor logic executes on the ULP processor core 110. However, the host controller 102 includes a wrapper associated with the always-on sensor logic of the ULP processor core 110.

[0028] Features associated with the location data domain 206 include location sensors and location logic. Example location sensors include GNSS sensors, Wi-Fi receivers, cellular network receivers, Bluetooth receivers, etc. In this example, both the driver for the location sensor and the location logic for analyzing and processing the location data generated by the location sensor are executed by the host controller 102.

[0029] Features associated with the visual data domain 208 include imaging hardware. In some examples, the imaging hardware includes an image signal processor (e.g., Figure 1 ISP 126) and camera. Figure 2 As shown in FIG, the drivers for the imaging hardware are located on the VCD offload engine 104. However, the host controller 102 includes a wrapper associated with the drivers contained in the VCD offload engine 104, enabling the host controller 102 to offload tasks associated with capturing images to the VCD offload engine 104. Features associated with the visual data domain 208 also include visual detection logic and visual recognition logic. In this example, the visual detection logic is located in the VCD offload engine 104 with a corresponding wrapper in the host controller 102, enabling object detection tasks to be offloaded to the VCD offload engine 104. In contrast, the visual recognition logic is associated with the host controller 102. However, the host controller 102 can offload digital signal processing and CNN acceleration to the VP offload engine 106.

[0030] Features associated with the audio data domain 210 include raw audio recognition logic and accurate audio recognition logic. In some examples, the raw audio recognition logic performs a course keyword recognition process to detect when the user speaks a keyword to wake up other functionality of the associated electronic device. In some examples, the raw audio recognition logic is implemented by utilizing the ULP processor core 110 so that it uses relatively small processing power to save power. Although the relatively small computational burden of the raw audio recognition logic can result in a relatively large error rate. However, in some examples, the audio logic is designed so that most errors are false positives rather than missed negatives. Once a keyword is detected using the raw audio recognition logic, the ULP processor core 110 wakes up or starts the audio offload engine 108 to implement accurate audio recognition logic together with DNN acceleration to confirm whether the raw audio recognition is correct or a false positive. In this example, the host controller 102 includes a wrapper associated with the accurate audio recognition logic contained in the audio offload engine 108.

[0031] Example features associated with the platform services domain 212 include pulse width modulation (PWM), which controls, for example, the blinking of a light emitting diode (LED); cryptographic security, for example, to authenticate dynamically loaded content in the firmware of the sensor hub 100; and Bluetooth Low Energy (BLE) communication. In this example, all platform services (whether hardware-based, firmware-based, and / or software-based) are supported within the host controller 102.

[0032] like Figure 2 As shown in Table 200 of FIGURE 2, all hardware drivers and all sensor data processing logic, except for the raw audio recognition logic, are either located within the host controller 102 or represented in the host controller 102 through a wrapper. As a result, all functionality provided by the context sensor hub 100 can be controlled, managed, and / or configured through the host controller 102. Therefore, the OEM does not need to directly code for any of the offload engines 104, 106, 108 or the ULP processor core 110, significantly simplifying the user experience for developers.

[0033] Cause and Figure 1 The example hardware architecture shown in FIG. Figure 3 The block diagram shows Figure 1 The example firmware architecture of the example context sensor hub 100 is shown in FIG. Although the VP offload engine 106 and the audio offload engine 108 are shown in the illustrated example, the details of their firmware structures have been omitted for simplicity. However, the firmware architecture of both the VP offload engine 106 and the audio offload engine 108 are substantially similar to the firmware architecture of the VCD offload engine 104. Although described in terms of firmware, Figure 3, but in some examples, Figure 3 One or more aspects of the invention may additionally or alternatively be implemented in hardware.

[0034] like Figure 3 As shown in the illustrated example, the host controller 102 includes basic infrastructure such as a board support package (BSP) 302, a boot read-only memory (ROM) 304, and a real-time operating system (RTOS) 306. Unlike known sensor hubs, the example context sensor hub 100 also includes an inter-process communication (IPC) interface 308 within the BSP 302 of the host controller 102. The IPC interface 308 enables the host controller 102 to communicate with the offload engines 104, 106, 108 and the ULP processor core 110 using IPC technology. The IPC technology can be hardware-centric. As a result, as with the context sensor hub 100, the IPC technology is not particularly well-suited as an application programming interface (API) for offloading specific computing tasks to different processor core circuits (e.g., different processing cores). Therefore, in some examples, the architecture of the host controller 102 includes an offload API proxy 310. The example API proxy 310 is based on IPC technology, but provides more meaningful API functionality for offloading different computing tasks. As an example, the host controller 102 may execute vision logic that calls the API agent 310 to offload certain image processing tasks (eg, image cropping, image scaling, etc.) to the VP DSP 128 of the VP offload engine 106. Figure 1 ). Additionally, the vision logic of the host controller 102 can call the API agent 310 a second time to open the cropped image and run the CNN model on the cropped image for object recognition. In some examples, the API agent 310 passes the request to the VP offload engine 106 to perform object recognition using the VP DSP 128 and the CNN accelerator 130. Passing commands from the host controller 102 through the API agent 310 to other cores of the context sensor hub 100 in this manner simplifies the developer's ability to configure, customize, and / or control the context sensor hub 100 because the developer only needs to communicate directly with the host controller 102, rather than with each specific core to be utilized.

[0035] like Figure 3 As shown in , the example architecture of the host controller 102 includes multiple sensor data sources 312. As used herein, a sensor data source corresponds to either: a sensor hardware driver associated with a physical sensor that outputs raw sensor data, and data processing logic that analyzes or processes the sensor data. Figure 3In the example shown in the figure, the sensor data source 312 includes a sensor hardware driver 314 and a sensor logic 316, either or both of which can be implemented in firmware. Unlike traditional sensor hubs that are limited to relatively specific uses, the example context sensor hub 100 of this example also includes the ability to aggregate and process context-based information. Therefore, in this example, other sensor data sources 312 in the host controller 102 include position logic 318, audio logic 320, and vision logic 322. As described above in conjunction with Figure 2 As mentioned, some sensor data sources 312 may be executed directly in the host controller 102. However, in some examples, at least some of the sensor data sources 312 represented in the host controller 102 may be wrappers with actual drivers and / or sensor data processing logic that offload one or more of the engines 104, 106, 108 and / or the ULP processor core 110.

[0036] The example host controller 102 includes a context framework 324 that serves as a runtime environment for the sensor data source 312 (e.g., sensor hardware driver 314, sensor logic 316, location logic 318, audio logic 320, and / or vision logic 322) and handles the data flow and control flow of the sensor data source 312. The context framework also handles the data flow and control flow to and from the host interface 326. The host interface 326 interfaces with the application processor (AP) of the associated electronic device (e.g., smartphone, tablet, PC, etc.) to support operating system (OS) integration. As described more fully below, in some examples, the context sensor hub 100 can be implemented as a standalone processor (rather than being implemented as a coprocessor of the AP). In some such examples, the host interface 326 can be omitted.

[0037] Compared to the typical framework of a traditional sensor hub, the example context framework 324 includes several differences and additional capabilities. As already mentioned, rather than only providing an API for basic sensor data (e.g., sensor hardware drivers 314 and sensor logic 316), the context framework 324 also provides extensions to the API support for context-based location logic, audio logic, and vision logic 318, 320, 322. The API enables the location logic, audio logic, and vision logic to be viewed and / or conceptualized as sensors (e.g., virtual sensors) in a manner similar to how a traditional sensor hub treats sensor hardware drivers (e.g., physical sensors) and associated sensor logic. In addition, the API enables wrappers for drivers and / or logic located on separate cores to be viewed and / or conceptualized as sensors in a manner similar to how a traditional sensor hub treats sensor hardware drivers (e.g., physical sensors) and associated sensor logic.

[0038] Many known sensor hubs run all sensor firmware in a single thread. In contrast, in some examples, the context framework 324 disclosed herein supports multiple threads to enable individual sensor data processing to occur simultaneously. This can be important when performing computationally intensive operations (e.g., visual detection and / or recognition), which can take relatively long periods of time compared to simpler operations, thereby causing delays when only a single thread is present.

[0039] Unlike many known sensor hubs, the example context framework 324 of the context sensor hub 100 provides customization and / or configurability of sensor parameters. That is, rather than building a sensor hub with a fixed configuration, the context framework 324 enables developers to flexibly define and / or adjust sensor parameters for specific applications. This is important because some context-based sensors (e.g., cameras) and / or sensor data logic (e.g., voice detection) include several configuration parameters (e.g., camera resolution, audio sampling frequency, etc.), which should be variable based on specific situations to achieve better performance.

[0040] In some examples, the execution of some of the more computationally intensive, context-based processing logic uses larger blocks of reference data. For example, location logic may involve referencing a Wi-Fi fingerprint database to determine location based on Wi-Fi signals. As another example, vision logic for facial recognition may involve referencing a database of user features for facial recognition. In some examples, because local memory (e.g., shared memory 116) may be insufficient, the context framework 324 enables such larger blocks of data to be loaded from the AP (e.g., via the host interface 326) and / or written back to the AP.

[0041] Context-based processing logic associated with visual detection and / or recognition and / or associated with audio detection and / or recognition is only as accurate as the artificial intelligence (AI) model used when performing the analysis. However, in examples with limited shared memory 116 (e.g., limited to 10.5MB in some examples) and no direct access to the system memory of the associated electronic device, the data processing model is relatively limited. The example context framework 324 enables dynamic updating of models to expand the capabilities of AI analysis. For example, with 1MB of available memory, the automatic speech recognition process can use a model with 50 keywords (corresponding to approximately 922KB of memory). In some examples, the context framework 324 enables different models containing different 50 keywords to be dynamically loaded from the AP to the shared memory 116 of the context sensor hub 100 to update or replace the first model, thereby effectively expanding the number of recognizable keywords to 100 without increasing the memory size. In some examples, the specific model used at any given moment can depend on the situation determined based on the analysis of other sensor data. For example, when a person is detected in a car, a first speech recognition model with keywords expected when the person is driving can be loaded, while when a person is detected at home, a second speech recognition model with keywords expected when the person is at home can be loaded. As another example, an object recognition model for 20 objects is approximately 1.8MB. To save memory size, in some examples, the 20 objects can be divided into two or more categories (e.g., indoor objects vs. outdoor objects) to form multiple smaller models with fewer objects, which can be individually selected for dynamic loading when corresponding to objects of a specific category (based on the detected situation).

[0042] As mentioned above, the example context framework 324 manages and / or controls the data flow and control flow of the sensor data sources 312 (e.g., all data sources), regardless of whether the driver and / or logic associated with a particular sensor data source 312 is located at the host controller 102 or on a different core (but represented by a wrapper in the host controller 102). To manage the offloading of computational tasks and the proper flow of data between the different cores and the associated sensor data sources, the example context framework 324 maintains the following data structures: which define the control and data flow relationships between the different sensor data sources and the location (e.g., a specific core) of the driver and / or logic for each sensor data source. The example sensor tree 400 representing such relationships is shown in FIG. Figure 4 Shown in.

[0043] As mentioned above, the example context framework 324 enables sensor data processing logic to be represented or conceptualized as sensors. To distinguish the data processing logic that processes the sensor data output by the hardware sensors from the drivers that enable the hardware sensors to collect that output data, each data processing logic represented in the sensor tree 400 is referred to as a "virtual" sensor, while the drivers are referred to as "physical" sensors. Thus, the virtual sensors and physical sensors shown in the sensor tree 400 correspond to the sensor data sources 312 provided in the architecture of the host controller 102. Figure 4 In the example shown in FIG, the sensor tree 400 represents two hardware drivers or physical sensors (eg, physical sensor A 402 and physical sensor B 404) and four data processing logic or virtual sensor items (eg, virtual sensor A 406, virtual sensor B 408, virtual sensor C 410, virtual sensor D 412).

[0044] In addition to identification of individual sensor data sources (whether virtual or physical), the example sensor tree 400 also includes identifiers indicating the specific core where the associated sensor data source (driver or data processing logic) is located. Figure 4 In the example sensor tree 400, physical sensor A, physical sensor B, and virtual sensor D are all located on the VP offload engine 106; virtual sensor A is located on the ULP processor core 110; virtual sensor B 408 is located on the VCD offload engine 104; and virtual sensor C 410 is located on the host controller 102. The virtual sensors may be software that executes on the corresponding core or on hardware circuits within the corresponding core to perform corresponding functions.

[0045] exist Figure 4 In the example shown in the figure, arrows represent the data flow between different sensors. In other words, the arrows represent the client-server relationship between different sensors. For example, the results of executing virtual sensor A 406 are provided to virtual sensor B 408. Therefore, virtual sensor A 406 is a server or data provider for virtual sensor B 408, which is a client of virtual sensor A. The results of virtual sensor B 408 are in turn provided to virtual sensor D 412, which in turn provides the output results to virtual sensor C 410. Furthermore, the output of physical sensor A 402 is provided to virtual sensor C 410, and the output of physical sensor B 404 is provided to virtual sensor D 412.

[0046] In some examples, the context framework 324 maintains the sensor tree 400 and / or any suitable data structure representing identifiers, tags, and relationships in the sensor tree 400 to control and / or manage the operation of the context sensor hub 100. For example, the designation of a particular core as corresponding to a particular sensor enables the example context framework 324 to determine when the host controller 102 and / or the different cores may be inactive, entering a sleep mode. In particular, Figure 4 In the example sensor tree 400 of FIG, virtual sensor C 410 is the only sensor actually implemented in the host controller 102. Therefore, if virtual sensor C 410 is on and running (e.g., the associated sensor data processing logic is currently processing sensor data), then the host controller 102 must be on or active. On the other hand, if virtual sensor C 410 is not running (although it may be on (e.g., waiting for an event result from virtual sensor D)), then the context framework 324 checks whether any other sensors in the sensor tree 400 are currently running on the host controller 102. If none of the running sensors are marked as being located in the host controller 102, then the context framework 324 puts the host controller 102 to sleep.

[0047] exist Figure 4The data flows represented in sensor tree 400 also define when to wake a particular core from an inactive state or sleep mode. For example, as mentioned above, because the output of physical sensor A 402 is provided to virtual sensor C 410, virtual sensor C 410 (executing on host controller 102) is a client of physical sensor A 402. Therefore, in some examples, physical sensor A 402 and / or its output are defined as events that trigger the host controller 102 to wake up, so that virtual sensor C 410 can receive and process the output. In example sensor tree 400, physical sensor B 404 does not include any clients identified as being located on host controller 102. The only client of physical sensor B 404 is virtual sensor D 412. Both sensor B 404 and sensor D 412 are located on VP offload engine 106. Therefore, events associated with the output of physical sensor B 404 do not trigger the host controller 102 to wake up. Instead, the result of executing virtual sensor B 408 is passed directly to virtual sensor D 412 within the VP offload engine 106. Virtual sensor A 406 has a single client defined in the example sensor tree 400 corresponding to virtual sensor B 408. In the event that both virtual sensors 406, 408 reside in different cores, the output of virtual sensor A 406 is configured to trigger an event that causes the VCD offload engine 104 to be activated to execute virtual sensor B. However, because neither virtual sensor A 406 nor virtual sensor B 408 resides on the host controller 102, there is no need to wake up or activate the host controller 102.

[0048] Return to reference Figure 3In the illustrated example, the architecture of the host controller 102 includes a Java runtime environment 328 to enable always-on applications corresponding to Java applets 330. In the illustrated example, all aspects implemented in the host controller 102 (except the Java runtime environment 328 and the Java applets 330) are implemented in the C programming language. Accordingly, in this example, the host controller 102 includes a C-Java agent 332 to enable interaction between the Java runtime environment 328 and the context framework 324. In this way, the Java applets 330 can open, close, configure, stream, and / or otherwise access any content within the context sensor hub 100. In some examples, all aspects of the context sensor hub 100 (except the Java applets 330) are statically built. In contrast, the Java applets 330 can be loaded and unloaded dynamically. If the Java applets contain malware, this can pose a potential security threat. Accordingly, in some examples, the Java runtime environment 328 performs signature verification on Java applets loaded or unloaded to the host controller 102 to prevent unauthorized applications. In addition, the Java runtime environment 328 provides permission checking, memory isolation, and sandboxing for Java applets 330. Thus, the Java runtime environment 328 can provide more flexibility and simplicity in configuring and running the context sensor hub 100 than would be possible if the hub were limited to static firmware.

[0049] The example Java applet 330 is an application that can perform any suitable function in the always-on mode. That is, the Java applet 330 can be triggered and / or executed when the associated electronic device is in a sleep mode or an idle mode. As a specific example, in response to the VP offload engine 106 recognizing a specific gesture of the user, the Java applet 330 can use BLE signals to communicate with the user's smart home gateway to turn on the lights in the user's home or perform some other operation. Although Figure 3 A Java runtime environment 328 and a Java applet 330 are shown in FIG, but any other runtime environment may alternatively be implemented to enable execution of the corresponding applications.

[0050] Figure 3 The example architecture of the host controller 102 shown in FIG includes platform services 334, which include any type of functionality or service that can be accessed and / or used by the Java applet 330 and / or other logic (e.g., sensor data processing logic). Example services include PWM control, encryption services, BLE stack, etc.

[0051] For simplicity, Figure 3 An example architecture of the ULP processor core 110 is shown in less detail than the host controller 102. As shown in the illustrated example, the architecture of the ULP processor core 110 includes, for example, a basic infrastructure 336 that includes infrastructure components such as a BSP, an RTOS, and a scheduler. The example architecture of the ULP processor core 110 also includes an example ULP service framework 338. The ULP service framework 338 has a purpose and function similar to the context framework 324 in the host controller 102, except that the ULP service framework 338 is limited to managing the ULP processor core 110. More specifically, in some examples, the ULP service framework 338 controls and / or manages sensor hardware drivers 340 and / or data processing logic 342 contained in the ULP processor core 110. As described above in conjunction with Figure 2 As mentioned, in some examples, the drivers 340 correspond to sensors that are always running (e.g., accelerometers, gyroscopes, microphones, etc.), and the logic 342 corresponds to logic that will always run (e.g., basic sensor logic and / or raw audio recognition logic). The example ULP processor core 110 includes an IPC interface 344 to enable communication with the host controller 102 via the IPC interface 308 of the host controller 102. In some examples, the context framework 324 in the host controller 102 provides configuration and / or control data to the ULP service framework 338 to facilitate the ULP service framework 338 to perform its appropriate functions. For example, in some examples, the context framework 324 provides the sensor tree 400 or other corresponding data structure to the ULP service framework 338 to enable the ULP service framework 338 to determine when outputs associated with the hardware drivers 340 and / or logic 342 will be used as events to trigger the launch of different cores.

[0052] The example architecture of the VCD offload engine 104 is substantially similar to the architecture of the ULP processor core 110. Figure 3, the VCD offload engine 104 includes a base infrastructure 346, an offload services framework 348 (similar to the ULP services framework 338 described above), data processing logic 350, sensor hardware drivers 352, and an IPC interface 354. The architectural difference between the ULP processor core 110 and the VCD offload engine 104 is that the logic 350 in the VCD offload engine 104 performs different computational tasks than the logic 342 in the ULP processor core 110. Similarly, the drivers 352 in the VCD offload engine 104 are associated with different hardware than the drivers 340 in the ULP processor core 110. Another difference between the ULP processor core 110 and the VCD offload engine 104 is that the VCD offload engine 104 includes one or more libraries 356 used by the corresponding logic 350.

[0053] exist Figure 3 Not shown are example architectures for the VP offload engine 106 and the audio offload engine 108. However, in some examples, the architecture of these offload engines 106 is similar to that shown for the VCD offload engine 104, except that different hardware drivers, logic, and associated libraries are implemented in each architecture.

[0054] Although Figure 3 The figure shows the implementation Figure 1 The context sensor hub 100 is an example embodiment of the present invention, but may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. Figure 3 One or more of the elements, processes and / or devices illustrated in FIG. Figure 1The example host controller 102 (including an example BSP 302, an example boot ROM 304, an example RTOS 306, an example IPC interface 308, an example offload API agent 310, an example sensor data source 312 (including an example sensor hardware driver 314, an example sensor logic 316, an example position logic 318, an example audio logic 320, and / or an example vision logic 322), an example context framework 324, an example host interface 326, an example Java runtime environment 328, an example Java applet 330, an example C-Java agent 332, and / or an example platform service 334), an example ULP processor core 110 (including an example basic infrastructure 336, an example ULP service framework 338, example sensor hardware driver 340, example logic 342, and / or example IPC interface 344), example VCD offload engine 104 (including example basic infrastructure 346, example offload service framework 348, example logic 350, example sensor hardware driver 352, example IPC interface 354, and / or example library 356), example VP offload engine 106, example audio offload engine 108 and / or more generally, the example context sensor hub 100 may be implemented by hardware, software, firmware and / or any combination of hardware, software and / or firmware.Thus, for example, the example host controller 102 (including the example BSP 302, the example boot ROM 304, the example RTOS 306, the example IPC interface 308, the example offload API agent 310, the example sensor data source 312 (including the example sensor hardware driver 314, the example sensor logic 316, the example location logic 318, the example audio logic 320, and / or the example vision logic 322), the example context framework 324, the example host interface 326, the example Java runtime environment 328, the example Java applet 330, the example C-Java agent 332, and / or the example platform services 334), the example ULP processor core 110 (including the example basic infrastructure 336, the example ULP service framework 338, the example sensor hardware driver 340, the example logic 342, and / or the example IPC interface 344), the example VCD offload engine 104 (including the example basic infrastructure 346, the example offload services The example context sensor hub 100 may be implemented as any of the framework 348, the example logic 350, the example sensor hardware driver 352, the example IPC interface 354, and / or the example library 356), the example VP offload engine 106, the example audio offload engine 108, and / or more generally, the example context sensor hub 100 may be implemented by one or more analog or digital circuits, logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPU(s), digital signal processor(s) (DSP(s), application specific integrated circuit(s) (ASIC(s), programmable logic device(s) (PLD(s), and / or field programmable logic device(s) (FPLD(s)).When any of the apparatus or system claims of this patent are read to encompass pure software and / or firmware implementations, the example host controller 102 (including an example BSP 302, an example boot ROM 304, an example RTOS 306, an example IPC interface 308, an example offload API agent 310, an example sensor data source 312 (including an example sensor hardware driver 314, an example sensor logic 316, an example position logic 318, an example audio logic 320, and / or an example vision logic 322), an example context framework 324, an example host interface 326, an example Java runtime environment 328, an example Java applet 330, an example C-Java agent 332, and / or an example platform services 334), an example ULP processor core 110 (including an example basic infrastructure 336, an example ULP service framework 338, an example ULP server framework 339, an example server framework 340, an example server framework 341, an example server framework 342, an example server framework 343, an example server framework 344, an example server framework 345, an example server framework 346, an example server framework 347, an example server framework 348, an example server framework 349 ... At least one of the example sensor hardware driver 340, the example logic 342, and / or the example IPC interface 344), the example VCD offload engine 104 (including the example basic infrastructure 346, the example offload service framework 348, the example logic 350, the example sensor hardware driver 352, the example IPC interface 354, and / or the example library 356), the example VP offload engine 106, and / or the example audio offload engine 108 is hereby expressly defined as comprising a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disc, etc., containing software and / or firmware. Further, Figure 1 An example context sensor hub 100 may include one or more elements, processes, and / or devices (in addition to Figure 3 components, processes and / or devices other than or in lieu of those illustrated in the drawings Figure 3 As used herein, the phrase "in communication" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but also includes selective communication at periodic intervals, predetermined intervals, non-periodic intervals, and / or one-time events.

[0055] Figures 5 to 7 The diagram shows Figure 3FIGURE 5 illustrates example phases of operation for implementing an always-on facial recognition process using an example architecture. In this particular example, the first phase of operation 502 begins when the sensor data processing logic 342 in the ULP processor core 110 detects a trigger event, such as a pick gesture identified based on motion data from an accelerometer and / or gyroscope. In this example, the ULP processor core 110 wakes up the host controller via the corresponding host controller 102 and the IPC interfaces 308, 344 in the ULP processor core 110. The context framework receives the event data and passes it to the vision logic 322 corresponding to facial recognition. In the second phase of operation 504, the vision logic 322 begins running on the host controller 102 and invokes the camera to be opened via the context framework 324. Consequently, the context framework 324 opens the camera via the VCD offload engine 104, where the camera's hardware driver 352 resides. At this point, the camera begins streaming image data. In a third stage 506 of operation, for a captured image, the vision logic 322 calls the offload API proxy 310 to offload image pre-processing to pre-process the image by downscaling via the VP offload engine 106. In some examples, the call to the API proxy includes an address in the shared memory where the unprocessed image is to be stored and a second address in the shared memory where the processed image is to be stored.

[0056] Steering Figure 6 In the fourth stage 602 of operation, the vision logic 322 retrieves the pre-processed image from the shared memory and then calls the offload API agent 310 to offload the facial recognition analysis to the CNN hardware accelerator 130 associated with the VP offload engine 106 ( Figure 1 ). In some examples, the call to the API proxy includes the address where the CNN model is stored in the shared memory 116. The VP offload engine 106 implements a CNN accelerator to process the CNN model and stores the results in the shared memory 116. In the fifth stage 604 of operation, the vision logic 322 further processes the results of the CNN processing to generate a final output of the vision logic. Thereafter, in the sixth stage 606 of operation, the vision logic 322 passes the final result to the context framework 324, which passes the result to the Java runtime environment 328 via the C-Java proxy 332. In this example, the seventh stage or operation 608 involves the context framework 324 also passing the final result of the vision logic 322 to the host interface 326 to notify the application processor for further processing.

[0057] Steering Figure 7At the eighth stage 702 of operation, the Java runtime environment 328 notifies the Java applet 330 of the results of the facial recognition process performed by the vision logic 322. At the ninth stage 704 of operation, the Java applet 330 analyzes the results of the facial recognition analysis to, for example, determine whether the recognized face corresponds to a user with specific preferences. If so, then at the tenth stage 706 of operation, the Java applet 330 calls a platform service 334 (e.g., PWM) to control a peripheral device (e.g., flash an LED light).

[0058] Indicates the implementation Figure 1 and / or Figure 3 Example hardware logic, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof of the context sensor hub 100 are shown in a flowchart. Figures 8 to 15 Machine-readable instructions can be generated by a computer processor (such as Figure 1 and / or Figure 3 The example context sensor hub 100 may be an executable program or portion of an executable program executed by one or more of the processor cores of the context sensor hub 100. The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or memory associated with the context sensor hub 100, but alternatively, the entire program and / or a portion thereof may be executed by a device other than the context sensor hub 100 and / or embodied in firmware or dedicated hardware. Additionally, although reference is made to Figures 8 to 15 The flowchart shown in the figure describes an example program, but alternatively, many other methods of implementing the example context sensor hub 100 can be used. For example, the order of execution of the blocks can be changed, and / or some of the blocks described can be changed, eliminated, or combined. Additionally or alternatively, any or all of these blocks can be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware.

[0059] As mentioned above, Figures 8 to 15The example processes of can be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium (such as a hard drive, flash memory, read-only memory, optical disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk) in which information is stored for any duration (e.g., for an extended period of time, permanently, transiently, temporarily buffered, and / or cached information). As used herein, the term "non-transitory computer readable medium" is expressly defined to include any type of computer readable storage device and / or storage disk and to exclude propagating signals and to exclude transmission media.

[0060] As used herein, "including" and "comprising" (and all forms and tenses thereof) are used as open-ended terms. Thus, whenever a claim, in preamble form or within any kind of claim recitation, employs any form of "include" or "comprise" (e.g., comprises, includes, comprising, including, having, etc.), it is understood that additional elements, terms, etc. may be present without exceeding the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transition term, such as in the preamble of a claim, it is intended to be open-ended in the same manner as the terms "comprising" and "including." The term "and / or," for example, when used in a form such as A, B, and / or C, refers to any combination or subset of A, B, and C, such as (1) A only; 2) B only; (3) C only; (4) A and B; (5) A and C; (6) B and C; and (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase "at least one of A and B" is intended to refer to embodiments that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase "at least one of A or B" is intended to refer to embodiments that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein in the context of describing the performance or execution of processes, instructions, actions, activities, and / or steps, the phrase "at least one of A and B" is intended to refer to embodiments that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing the performance or execution of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to refer to an embodiment that includes any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0061] Figure 8The process begins in block 802 where the example ULP processor core 110 monitors the sensors that are always running. In block 804, the example ULP processor core 110 determines whether a new event has been triggered based on the sensor output. If not, then control returns to block 802. If so, then control proceeds to block 806 where the example ULP service framework 338 of the ULP processor core 110 determines whether the event has triggered a sensor data source. In some examples, the ULP service framework 338 determines whether the event has triggered a sensor data source by referencing a sensor tree (e.g., Figure 4 This determination is made by identifying which clients (physical sensor data sources or virtual sensor data sources) will receive data associated with the triggered event. As discussed above, the sensor data sources include sensor hardware drivers (in Figure 4 sensor tree 400 as physical sensors) or data processing logic (in Figure 4 ). If the example ULP service framework 338 determines that the event triggered the sensor data source (box 806), then control proceeds to box 808, in which the example ULP processor core 110 determines whether the core executing the sensor data source has been awakened. If not, then in box 810, the example ULP processor core 110 wakes up the relevant core before controlling the process to proceed to box 812. If the relevant core has been awakened, then control proceeds directly to box 812, in which the example ULP processor core passes the event data to the sensor data source. In some examples, the event data is passed based on a call to the offload API agent 310 to offload tasks for specific logic in the executing process. In box 814, the relevant core executes the sensor data source. That is, the relevant core implements the sensor hardware driver or data processing logic corresponding to the triggered sensor data source. The specific core that implements box 814 and the associated process depends on the specific sensor data source and its associated purpose. Figures 9 to 15 An example implementation of block 814 is provided. Thereafter, control proceeds to block 816. Returning to block 806, if the example ULP service framework 338 determines that the event did not trigger a sensor data source, control proceeds directly to block 816.

[0062] In block 816, the example ULP service framework 338 determines whether the event triggered the operation of a user-loaded application (e.g., the Java applet 330). If the example ULP service framework 338 determines that the event triggered the operation of a user-loaded application, control proceeds to block 818, where the ULP service framework 338 passes the event data to the runtime environment (e.g., the Java runtime environment 328) in the host controller 102 via the context framework 324. In block 820, the host controller 102 executes the application based on the event data. Thereafter, control proceeds to block 822. If the example ULP service framework 338 determines that the event did not trigger the operation of a user-loaded application (block 816), control proceeds directly to block 822.

[0063] In block 822, the example context sensor hub 100 determines whether at least one sensor data source (e.g., a sensor hardware driver or data processing logic) is running on a particular core. In some examples, a framework associated with each core (e.g., the context framework 324 for the host controller 102, the ULP services framework 338 for the ULP processor core 110, or the offload services framework for the corresponding offload engine 104, 106, 108) performs this determination of the sensor data source executed by its corresponding core. If no sensor data source is running for the particular core, control proceeds to block 824, where the corresponding framework (e.g., the context framework 324, the ULP services framework 338, or the offload services framework 348) puts the particular core into hibernation. Thereafter, control proceeds to block 826. If at least one sensor data source associated with the particular core is running, control proceeds directly to block 826. In block 826, the context sensor hub 100 determines whether another core exists. If so, control returns to block 822 to analyze the sensor data source for the next core. As mentioned above, in some examples, each of the cores performs the determination in block 822 separately so that the process may not loop as represented by block 826 .

[0064] In block 828, the particular kernel that executed the sensor data source in block 814 determines whether the execution of the sensor data source has triggered another event. If so, control returns to block 806 to repeat the process except for blocks 806 to 812, 816, and 818, which are now executed by the particular kernel (which is now executing the sensor data source that was triggered in block 806 in the previous iteration of the process). If it is determined that the output of the sensor data source has not triggered another event, control advances to block 830, in which the context sensor hub 100 determines whether to continue. If so, control returns to block 802. Otherwise, Figure 8 The example process ends.

[0065] As mentioned above, Figures 9 to 15 An example implementation of block 814 associated with different sensor data sources executed by different cores is provided. For example, Figure 9 The diagram illustrates the process of executing data processing logic in the ULP processor core 110 to count the number of steps of a user (ie, provide pedometer functionality). Figure 9 The example process begins in block 902, where the ULP processor core 110 analyzes the output of the always-on sensor based on data processing logic. In block 904, the example ULP processor core 110 determines whether the output indicates a step count. If so, control passes to block 906, where the example ULP processor core 110 increments the pedometer. Thereafter, Figure 9 The example process ends and returns to complete Figure 8 If the output does not indicate the number of steps, then Figure 9 The example process ends without incrementing the counter and returns to completion. Figure 8 process.

[0066] Figure 10 The diagram shows the data processing logic (e.g., in conjunction with the ULP processor core 110) for detecting speech in audio. Figure 2 The original speech recognition logic described in

[15] . Figure 10 The example process of the example begins in block 1002, where the ULP processor core 110 implements raw speech recognition logic based on a speech detection model. In block 1004, the example ULP processor core 110 stores the results in shared memory. Thereafter, Figure 10 The example process ends and returns to complete Figure 8 process.

[0067] Figure 11 Figure 1 shows the data processing logic (e.g., in conjunction with the audio offload engine 108) for detecting speech in audio. Figure 2 accurate speech recognition logic as described). Figure 11 The example process of the example audio offload engine 108 begins in block 1102, where the example audio offload engine 108 starts the DNN hardware accelerator 134. In block 1104, the example audio offload engine 108 implements the original speech recognition logic based on the speech detection model. In block 1106, the example audio offload engine 108 stores the results in the shared memory. Thereafter, Figure 11 The example process ends and returns to complete Figure 8 process.

[0068] Figure 12The diagram illustrates data processing logic used to determine position in host controller 102. In some examples, this process is triggered based on the output of logic executed on ULP processor core 110 that analyzes output from an always-on motion sensor. Figure 12 The example process begins in block 1202, where the host controller 102 activates (e.g., by executing an associated driver in the host controller 102) the position sensor(s). In block 1204, the example host controller 102 determines whether position data has been received from the position sensor(s). If not, then control remains in block 1204. If yes, then control proceeds to block 1206, where the example host controller 102 determines a position based on the position data from the position sensor(s). In block 1208, the example host controller 102 stores the result in shared memory. In block 1210, the example host controller 102 determines whether there is more position data (e.g., indicating that the electronic device is still moving). If yes, then control returns to block 1206. Otherwise, Figure 12 The example process ends and returns to complete Figure 8 process.

[0069] Figure 13 The diagram illustrates the sensor hardware driver used in the VCD offload engine 104 to capture images. Figure 13 The example process begins in block 1302, where the VCD offload engine 104 activates the camera sensor to capture an image. In block 1304, the example VCD offload engine 104 activates the camera ISP. In block 1306, the example VCD offload engine 104 processes the captured image using the camera ISP. In block 1308, the example VCD offload engine 104 stores the result in shared memory. Thereafter, Figure 13 The example process ends and returns to complete Figure 8 process.

[0070] Figure 14 The diagram illustrates the data processing logic used in the VCD offload engine 104 to detect objects in an image. Figure 14 The example process of begins in block 1402, where the VCD offload engine 104 performs visual detection analysis based on the object detection model. In block 1404, the example VCD offload engine 104 stores the results in a shared memory. Thereafter, Figure 14 The example process ends and returns to complete Figure 8 process.

[0071] Figure 15The diagram illustrates data processing logic in host controller 102 for identifying an object in an image. In some examples, the object may be a face. Figure 15 The example process begins in block 1502, where the example host controller 102 calls the API agent 310 to cause the VP offload engine 106 to perform object recognition using the CNN accelerator 130. In block 1504, the example host controller 102 post-processes the results of the object recognition. In block 1506, the example host controller 102 stores the final output in the shared memory 116. Thereafter, Figure 15 The example process ends and returns to complete Figure 8 process.

[0072] From the foregoing, it will be appreciated that example methods, apparatus, and articles of manufacture have been disclosed that enable aggregation and processing of both basic sensor data and more advanced context-based sensor data associated with, for example, location, vision, and / or audio analysis in an energy-efficient manner suitable for always-on applications. Context-based processing of relatively compute-intensive processes in an energy-efficient manner is achieved by integrating multiple processor cores into a single system (e.g., a single chip) so that specific cores can be started and powered when specific computational tasks need to be offloaded and then put to sleep when power conservation is not required. Management of functionality distributed across multiple different processor cores is facilitated by implementing one core as a host that contains logic or wrappers for computational tasks to access logic executed in other cores. Thus, the disclosed methods, apparatus, and articles of manufacture relate to one or more improvements in computer functionality.

[0073] Example 1 includes an apparatus comprising: a first processor core for implementing a host controller; and a second processor core for implementing an offload engine, the host controller comprising first logic for processing sensor data associated with an electronic device when the electronic device is in a low power mode, the host controller for offloading computing tasks associated with the sensor data to the offload engine, the offload engine comprising second logic for performing the computing tasks.

[0074] Example 2 includes an apparatus according to Example 1, further comprising a third processor core configured to operate in a lower power domain than either the first processor core or the second processor core, the third processor core configured to monitor sensors associated with the electronic device while the first processor core and the second processor core are asleep.

[0075] Example 3 includes the apparatus of Example 2, wherein the third processor core is to detect a trigger event based on sensor data output by the sensor and to wake up the host controller in response to the trigger event.

[0076] Example 4 includes the apparatus of any of Examples 2 or 3, wherein the host controller includes a hardware driver for the sensor.

[0077] Example 5 includes the apparatus of any of Examples 1 to 4, wherein the host controller includes a wrapper associated with the second logic.

[0078] Example 6 includes an apparatus according to any one of Examples 1 to 5, wherein the offload engine is a first offload engine, and the apparatus further includes a third processor core for implementing a second offload engine, wherein the first offload engine is used to implement a first sensor data analysis, and the second offload engine is used to implement a second sensor data analysis, and the first sensor data analysis is different from the second sensor data analysis.

[0079] Example 7 includes the apparatus of Example 6, wherein the first sensor data analysis corresponds to a visual detection analysis and the second sensor data analysis corresponds to a visual recognition analysis.

[0080] Example 8 includes the apparatus of Example 7, wherein the third processor core comprises a convolutional neural network accelerator.

[0081] Example 9 includes the apparatus of Example 8, wherein the host controller is to dynamically update the model to be analyzed using the convolutional neural network accelerator.

[0082] Example 10 includes an apparatus according to any one of Examples 6 to 9, wherein the first sensor data analysis corresponds to at least one of a visual detection analysis or a visual recognition analysis, and the second sensor data analysis corresponds to at least one of a speech detection analysis or a sound recognition analysis.

[0083] Example 11 includes the apparatus of Example 10, wherein the second processor core comprises a convolutional neural network accelerator and the third processor core comprises a deep neural network accelerator.

[0084] Example 12 includes the apparatus of any of Examples 10 or 11, wherein the host controller comprises third logic to determine a location of the electronic device.

[0085] Example 13 includes an apparatus according to any one of Examples 6 to 12, wherein the host controller is to store a data structure defining a client-server relationship between different sensor data sources, the different sensor data sources corresponding to any one of a sensor hardware driver or a sensor data processing logic, the data structure being used to identify which processor core of the first processor core, the second processor core, or the third processor core includes corresponding ones of the sensor data sources.

[0086] Example 14 includes the apparatus of Example 13, wherein the host controller is to determine when no sensor data source of the host controller is operating and is to put the host controller to sleep based on the determination.

[0087] Example 15 includes an apparatus according to any of Examples 13 or 14, wherein the host controller is to share a data structure with the first offload engine and the second offload engine, and at least one of the host controller, the first offload engine, or the second offload engine is to determine whether to wake up a different one than at least one of the host controller, the first offload engine, or the second offload engine based on a client-server relationship between different sensor data sources.

[0088] Example 16 includes the apparatus of any of Examples 1 to 15, wherein the host controller and the offload engine are capable of communicating via inter-process communication.

[0089] Example 17 includes the apparatus of example 16, wherein the host controller is further for offloading computing tasks by invoking an application programming interface (API) agent associated with inter-process communication.

[0090] Example 18 includes the apparatus of any one of Examples 1 to 17, wherein the host controller comprises a runtime environment for executing user-loaded applications when the electronic device is in the low power mode.

[0091] Example 19 includes the apparatus of any of Examples 1 to 18, wherein at least one of the first processor core or the second processor core is to implement multithreading.

[0092] Example 20 includes an apparatus according to any one of Examples 1 to 20, further comprising a memory shared by the first processor core and the second processor core, the memory having a first memory size, and a host controller for communicating with an application processor of the electronic device to load a data block associated with a second memory size that is larger than the first memory size.

[0093] Example 21 includes a non-transitory computer-readable medium comprising instructions that, when executed, cause at least one system to perform at least the following operations: utilize first logic of a first processor core to process sensor data associated with an electronic device when the electronic device is in sleep mode; offload computing tasks associated with the sensor data to an offload engine implemented in a second processor core; and process the sensor data based on second logic of the second processor core to implement the computing tasks.

[0094] Example 22 includes a non-transitory computer-readable medium according to Example 21, wherein the instructions further cause at least one system to perform the following operations: monitor sensors associated with the electronic device while the first processor core and the second processor core are asleep based on third logic of a third processor core, the third processor core being configured to operate in a lower power domain than either the first processor core or the second processor core.

[0095] Example 23 includes a non-transitory computer-readable medium according to Example 22, wherein the instructions further cause the at least one system to perform the following operations: detecting, using a third processor core, a triggering event based on sensor data output by a sensor; and waking up the first processor core in response to the triggering event.

[0096] Example 24 includes the non-transitory computer-readable medium of any of Examples 22 or 23, wherein the instructions further cause at least one system to implement a hardware driver for the sensor.

[0097] Example 25 includes the non-transitory computer-readable medium of any of Examples 21 to 24, wherein the instructions further cause at least one system to implement a wrapper associated with the second logic.

[0098] Example 26 includes a non-transitory computer-readable medium according to any one of Examples 21 to 25, wherein the offload engine is a first offload engine, and the instructions further cause at least one system to perform the following operations: using the first offload engine to perform a first sensor data analysis; and using a second offload engine implemented by a third processor core to perform a second sensor data analysis, the first sensor data analysis being different from the second sensor data analysis.

[0099] Example 27 includes the non-transitory computer-readable medium of Example 26, wherein the first sensor data analysis corresponds to a visual detection analysis and the second sensor data analysis corresponds to a visual recognition analysis.

[0100] Example 28 includes a non-transitory computer-readable medium according to Example 27, wherein the instructions further cause at least one system to implement a convolutional neural network accelerator.

[0101] Example 29 includes a non-transitory computer-readable medium according to Example 28, wherein the instructions further cause the at least one system to dynamically update, using the first processor core, a model to be analyzed using the convolutional neural network accelerator.

[0102] Example 30 includes a non-transitory computer-readable medium according to any one of Examples 26 to 29, wherein the instructions further cause at least one system to perform the following operations: using the first processor core to save a data structure defining a client-server relationship between different sensor data sources, the different sensor data sources corresponding to any of sensor hardware drivers or sensor data processing logic, the data structure being used to identify which processor core of the first processor core, the second processor core, or the third processor core includes corresponding ones of the sensor data sources.

[0103] Example 31 includes a non-transitory computer-readable medium according to example 30, wherein the instructions further cause at least one system to: determine a time when no sensor data source of the first processor core is running; and deactivate the processor core based on the determination.

[0104] Example 32 includes a non-transitory computer-readable medium according to any of Examples 30 or 31, wherein the instructions further cause at least one system to perform the following operations: share a data structure with a second processor core and a third processor core; and at least one of the first processor core, the second processor core, or the third processor core determines whether to wake up a different one than at least one of the first processor core, the second processor core, or the third processor core based on a client-server relationship between different sensor data sources.

[0105] Example 33 includes the non-transitory computer-readable medium of any of Examples 21 to 32, wherein the instructions further cause the at least one system to communicate with the first processor core and the second processor core using inter-process communication.

[0106] Example 34 includes the non-transitory computer-readable medium according to example 33, wherein the instructions further cause at least one system to offload computing tasks by invoking an application programming interface (API) agent associated with inter-process communication.

[0107] Example 35 includes a non-transitory computer-readable medium according to any one of Examples 21 to 34, wherein the instructions further cause the at least one system to implement a runtime environment in the first processor core for executing user-loaded applications when the electronic device is in a sleep mode.

[0108] Example 36 includes a non-transitory computer-readable medium according to any one of Examples 21 to 35, wherein the instructions further cause at least one system to communicate with an application processor of the electronic device to load a data block associated with a first memory size that is larger than a second memory size associated with a local memory shared by the first processor core and the second processor core.

[0109] Example 37 includes a method comprising: processing sensor data associated with an electronic device when the electronic device is in an idle mode based on first logic of a first processor core; offloading a computing task associated with the sensor data to an offload engine implemented in a second processor core; and processing the sensor data based on second logic of the second processor core to implement the computing task.

[0110] Example 38 includes a method according to Example 37, further comprising: monitoring sensors associated with the electronic device while the first processor core and the second processor core are asleep based on third logic of a third processor core, the third processor core being configured to operate in a lower power domain than either the first processor core or the second processor core.

[0111] Example 39 includes the method of Example 38, further comprising: detecting, using the third processor core, a triggering event based on sensor data output by the sensor; and waking up the first processor core in response to the triggering event.

[0112] Example 40 includes a method according to any one of Examples 37 to 39, wherein the offload engine is a first offload engine, and the method further includes: utilizing the first offload engine to perform a first sensor data analysis; and utilizing a second offload engine implemented by a third processor core to perform a second sensor data analysis, the first sensor data analysis being different from the second sensor data analysis.

[0113] Example 41 includes a method according to Example 40, wherein the third processor core includes a convolutional neural network accelerator, and the method further includes: using the first processor core to dynamically update the model to be analyzed using the convolutional neural network accelerator.

[0114] Example 42 includes the method of any of Examples 40 or 41, further comprising determining, using the first processor core, a location of the electronic device based on the sensor data.

[0115] Example 43 includes a method according to any one of Examples 40 to 42, further comprising: using the first processor core to save a data structure defining a client-server relationship between different sensor data sources, the different sensor data sources corresponding to any one of the sensor hardware driver or the sensor data processing logic, the data structure being used to identify which processor core of the first processor core, the second processor core, or the third processor core includes corresponding ones of these sensor data sources.

[0116] Example 44 includes the method of Example 43, further comprising: determining a time when no sensor data source of the first processor core is running; and deactivating the first processor core based on the determination.

[0117] Example 45 includes a method according to any of Examples 43 or 44, comprising: sharing a data structure with a second processor core and a third processor core; and at least one of the first processor core, the second processor core, or the third processor core determining whether to wake up a different one than at least one of the first processor core, the second processor core, or the third processor core based on a client-server relationship between different sensor data sources.

[0118] Example 46 includes the method of any one of Examples 37 to 45, further comprising: implementing communication between the first processor core and the second processor core using inter-process communication.

[0119] Example 47 includes the method of example 46, wherein the offloading of the computing task comprises calling an application programming interface (API) agent associated with inter-process communication.

[0120] Example 48 includes the method of any one of Examples 37 to 47, further comprising: implementing a runtime environment in the first processor core for executing user-loaded applications when the electronic device is in an idle mode.

[0121] Example 49 includes the method of any one of Examples 37 to 48, wherein at least one of the first processor core or the second processor core is to implement multithreading.

[0122] Example 50 includes a method according to any one of Examples 37 to 49, further comprising: communicating with an application processor of the electronic device to load a data block associated with a first memory size, wherein the first memory size is larger than a second memory size associated with a local memory shared by the first processor core and the second processor core.

[0123] Although certain example methods, apparatus, and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the claims of this patent.

Claims

1. A sensor hub comprising: Image signal processor (ISP), used to process image data provided by the image sensor; a digital signal processor (DSP) for processing sensor data provided by at least one of an audio sensor or the image sensor; First hardware accelerator; a second hardware accelerator; as well as memory, wherein each of (a) the ISP, (b) the DSP, (c) the first hardware accelerator, (d) the second hardware accelerator, and (e) the memory is implemented as a system on a chip (SOC), and the sensor hub operates in an always-on state.

2. The sensor hub according to claim 1, wherein: At least one of the first hardware accelerator or the second hardware accelerator is used to perform computational tasks that are offloaded from a separate processor circuit.

3. The sensor hub according to claim 1, wherein: At least one of the first hardware accelerator or the second hardware accelerator is used to process artificial intelligence (AI) workloads.

4. The sensor hub according to claim 1, wherein: At least one of the ISP, the DSP, the first hardware accelerator, or the second hardware accelerator is configured to facilitate determining a context based on the sensor data, the determination of the context being based on artificial intelligence (AI) analysis.

5. The sensor hub according to claim 1, wherein: The sensor hub is used to perform at least one of sound recognition or voice detection.

6. The sensor hub according to claim 1, comprising: The interface circuit is used to communicate with a processor circuit outside the SOC.

7. The sensor hub according to claim 6, wherein: At least one of the ISP, the DSP, the first hardware accelerator, the second hardware accelerator, or the memory is in operation when a processor circuit external to the SOC is in at least one of a low power mode, an idle mode, or a sleep mode.

8. The sensor hub according to claim 7, wherein: The sensor hub is configured to wake up a processor circuit external to the SOC from the at least one of the low power mode, the idle mode, or the sleep mode.

9. A device comprising: Image signal processor (ISP), which processes the output of the image sensor; a digital signal processor (DSP) for processing at least one of an output of an audio sensor or an output of the image sensor; the first neural network accelerator; Second neural network accelerator; Memory; as well as An integrated sensor hub comprising each of (a) the ISP, (b) the DSP, (c) the first neural network accelerator, (d) the second neural network accelerator, and (e) the memory, wherein at least one of the ISP, the DSP, the first neural network accelerator, the second neural network accelerator, or the memory operates in an always-on state.

10. The device according to claim 9, wherein At least one of the first neural network accelerator or the second neural network accelerator is used to perform computational tasks offloaded from a separate processor circuit.

11. The device according to claim 9, wherein At least one of the first neural network accelerator or the second neural network accelerator is used to process artificial intelligence (AI) workloads.

12. The device according to claim 9, wherein At least one of the ISP, the DSP, the first neural network accelerator, or the second neural network accelerator is used to facilitate determining a context based on at least one of an output of the audio sensor or an output of the image sensor, wherein the determination of the context is based on artificial intelligence (AI) analysis.

13. The device according to claim 9, wherein The integrated sensor hub is used to perform at least one of sound recognition or voice detection.

14. The apparatus according to claim 9, comprising: An interface circuit is provided for communicating with a separate processor circuit external to the integrated sensor hub.

15. The device according to claim 14, wherein At least one of the ISP, the DSP, the first hardware accelerator, the second hardware accelerator, or the memory is in operation while the separate processor circuit is in at least one of a low power mode, an idle mode, or a sleep mode.

16. The apparatus according to claim 15, wherein, The integrated sensor hub is configured to wake the separate processor circuit from the at least one of the low power mode, the idle mode, or the sleep mode.

17. A sensor hub comprising: means for processing image data provided by an image sensor; means for processing sensor data provided by at least one of an audio sensor or an image sensor; A first acceleration device, configured to accelerate data processing; A first acceleration device, configured to accelerate data processing; as well as All of (a) the device for processing image data, (b) the device for processing sensor data, (c) the first acceleration device, (d) the second acceleration device, and (e) the storage device are implemented as a system on a chip (SOC), and the sensor hub operates in an always-on state.

18. The sensor hub according to claim 17, wherein: At least one of the first acceleration device or the second acceleration device is used to perform computing tasks offloaded from a separate processor circuit.

19. The sensor hub according to claim 17, wherein: At least one of the first acceleration device or the second acceleration device is used to process artificial intelligence (AI) workloads.

20. The sensor hub of claim 17, wherein: The device for processing image data, the device for processing sensor data, the first acceleration device or the second acceleration device is used to facilitate determining a context based on the sensor data, where the determination of the context is based on artificial intelligence (AI) analysis.

21. The sensor hub of claim 17, wherein: The sensor hub is used to perform at least one of sound recognition or voice detection.

22. The sensor hub of claim 17, comprising: Means for communicating with a processor circuit external to the SOC.

23. The sensor hub of claim 22, wherein: When the processor circuit outside the SOC is in at least one of a low power mode, an idle mode, or a sleep mode, at least one of the device for processing image data, the device for processing sensor data, the first acceleration device, the second acceleration device, or the storage device is in operation.

24. The sensor hub of claim 23, wherein the sensor hub is configured to wake a processor circuit external to the SOC from at least one of the low power mode, idle mode, or sleep mode.

Images