SECURE INSOLED AREAS

Abstract

Techniques are disclosed relating to securing hardware accelerators used by a data processing device. In some embodiments, a data processing device includes one or more processors configured to co-execute trusted and untrusted processes in an isolated manner, which involves implementing a secure environment in which a set of security criteria is enforced for data of the trusted processes.The data processing device further includes several heterogeneous hardware accelerators configured to implement isolated areas of the secure environment, extending the enforcement of one or more of the set of security criteria within the hardware accelerators for data distributed to the hardware accelerators for the performance of tasks assigned to the trusted processes.

Description

Technical field

[0001] This disclosure relates generally to data processing devices and in particular to improving the security of data processing devices. Description of the state of the art

[0002] Data processing devices, such as computers, mobile phones, tablets, and other devices, can often store large amounts of sensitive data. For example, a user's mobile phone might store contact information for friends and family, photos, text messages, emails, passwords, financial information, and so on. This sensitive data can also include various sensor data provided by sensors embedded in the devices, such as one or more cameras, microphones, location sensors, biometric sensors, health sensors, and so forth. To prevent unauthorized access to this data, devices can employ various security techniques. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram illustrating an example of a device that supports secure exclaves. Fig. Figure 2 is a block diagram illustrating an example of the device's software architecture. Fig. 3A to 3C are block diagrams that illustrate examples of data flows between sensors and applications of the device. Fig. Figure 4 is a representation illustrating an example of a display unit that implements a safe isolated area. Fig. Figure 5 is a representation illustrating an example of an image signal processor that implements a safe isolated area. Fig. Figure 6 is a representation illustrating an example of an audio unit that implements a secure isolated area. Fig. Figure 7 is a representation illustrating an example of a neural engine that implements a secure isolated area. Fig. Figure 8 is a representation illustrating an example of a graphics processing unit that implements a secure isolated area. Fig. 9A to 11B are flowcharts that illustrate examples of procedures implementing the functionality described herein. Fig. Figure 12 is a block diagram illustrating an exemplary data processing device implementing the functionality described herein. Fig. Figure 13 is a representation illustrating exemplary applications for systems and devices that implement the functionality described herein. Fig. Figure 14 is a block diagram illustrating an exemplary computer-readable medium that stores circuit design information for implementing devices that have the functionality described herein. DETAILED DESCRIPTION

[0003] The protection of sensitive data is traditionally the prerogative of 1) specific software (e.g., an operating system / kernel) running on a device's central processing unit (CPU), or 2) specialized hardware (sometimes referred to as "isolated areas") that employs strong separation techniques to prevent external entities (e.g., executing processes, the CPU, etc.) from directly accessing internal data. While these enforcement mechanisms can provide a secure environment for sensitive data, this secure environment is typically limited to a specific region of a device controlled by the enforcement mechanism and does not extend to other regions outside its control, which can represent a potential attack vector.For example, an operating system might be able to prevent a first process from accessing sensitive data of a second process while both processes are running on the CPU of a device—a region controlled by the operating system. However, the first process might be able to circumvent this enforcement by accessing components outside the CPU, such as the device's graphics processing unit (GPU) or other hardware accelerators.

[0004] The present disclosure describes embodiments in which a secure environment is extended by implementing secure isolated areas. As discussed in various embodiments, a data processing device may include one or more processors configured to co-execute trusted and untrusted processes in an isolated manner, which includes implementing a secure environment in which a set of security criteria is enforced for data of the trusted processes. The data processing device may include multiple heterogeneous hardware accelerators (e.g., GPUs, neural engines, peripheral accelerators, etc.).This includes hardware accelerators configured to implement isolated areas of the secure environment, extending the enforcement of one or more of the set of security criteria within the hardware accelerators for data distributed to them for performing tasks assigned to trusted processes. In many cases, extending a secure environment in this way can reduce the attack surface for gaining access to sensitive data. It can also allow larger hardware resources to be available for sensitive tasks when using unsecured resources is undesirable.

[0005] As an example discussed below in connection with some embodiments, a data processing device may include a sensor that collects sensitive data, such as a camera, microphone, etc. Sensor-processor switching logic of the data processing device may process this sensor data and negotiate one or more conditions under which an untrusted consumer is permitted to receive the processed sensor data. For example, access to the sensitive data may be permitted as long as a user is notified of the sensor's use. In some embodiments, a user interface hardware accelerator, implementing an isolated area of ​​the secure environment, processes received data to generate an output for a user interface of the data processing device, such as a display, speaker, etc.The hardware accelerator can receive a signal from a source in the secure environment indicating that the sensor has been activated and, before displaying the output via the user interface, insert an artifact / indicator (e.g., a pixel, a sound, etc.) of the activated component into the output. In some embodiments, the user interface pipeline switching logic can further extract data corresponding to the point where the indicator was inserted into the output and provide this data to a target in the secure environment for analysis to determine whether the indicator remains in the displayed output. If the indicator is not present in the output, the secure environment can notify an isolated area implemented by the sensor processor switching logic to prevent access to the sensor data.By using one or more isolated areas to protect the insertion and analysis of this indicator, the data processing device can make it more difficult for malicious software running outside the secure environment to activate the sensor and affect the corresponding user activity indicator.

[0006] Now with reference to Fig. Figure 1 shows a block diagram of a data processing device 10 that supports isolated areas. The device 10 can be attached to any suitable data processing device, such as a desktop computer, laptop, tablet, mobile phone, server, or any other device described below in relation to Fig. 13, discussed. In the illustrated embodiment, the device 10 includes one or more processor(s) 110, a memory 120, and heterogeneous hardware accelerators 130 coupled to one another via a fabric 102. The heterogeneous hardware accelerators 130 include a display unit 140, an image signal processor (ISP) 150, an audio unit 160, a neural engine 170, and a graphics unit 180. In other embodiments, the device 10 may be implemented differently than shown—for example, the device 10 may include different accelerators 130, additional components, as described in relation to Fig. 12 discussed, etc.

[0007] The processors 110 are processors (e.g., CPUs) configured to execute program instructions from various processes 112 to perform different operations. As shown, these processes 112 include trusted processes 112A, which may include processes traditionally associated with the kernel space, such as drivers, kernel services, etc. The trusted processes 112A may also include processes that work with sensitive data, such as user data, sensor data, etc. The processes 112 also include untrusted processes 112B, which may include processes traditionally associated with the user space, such as user applications, office suites, games, etc. In some embodiments, the trusted processes 112A are assumed to be more trustworthy than the untrusted processes 112B because the processes 112A are trusted by a trusted source (e.g.,Device manufacturers) can provide devices that have verifiable integrity (e.g., using digital signatures), can be designed to have a reduced attack surface, can be given a higher level of security protection, etc.

[0008] To improve the security of the device 10 in various embodiments, the processors 110 are configured to execute the trusted processes 112A and the untrusted processes 112B together in an isolated manner, which includes implementing a secure environment in which a set of security criteria is enforced. These criteria can define how trusted process data 122A is maintained for the trusted processes 112A (as well as untrusted process data 122B for the untrusted processes 112B in some embodiments). For example, the untrusted processes 112B can be prevented from accessing regions of memory 120 that store the trusted process data 122A.These criteria can also define which resources are permitted (and under which conditions) to access the trusted process data 122A and interact with processes 112A. These criteria can also define which resources (e.g., cameras, microphones, location sensors, motion sensors, health tracking sensors, etc.) are accessible to the untrusted processes 112B (and under which conditions). These criteria can define which execution permissions can be assigned to processes 112A and 112B. These criteria can also define the conditions / contexts under which the trusted process data 112A is permitted to flow from the secure environment to the untrusted processes 112B (or other untrusted destinations).

[0009] The 110 processors can implement the secure environment using any number of different techniques. In some embodiments, the 110 processors are configured to run the 112A processes in an authorized execution mode (Ring 2 or lower) that is not available to the untrusted 112B processes. The 110 processors can also partition memory 120 and prevent memory access by the untrusted 112B processes to regions not allocated to the secure environment. As with respect to Fig. As described in sections 2 to 3C, the 110 processors can further employ a software architecture in which the trusted processes 112A and the untrusted processes 112B run on separate operating systems, which can use a filtering layer to control the data flow between the 112 processes. In some embodiments, these operating systems also implement paravirtualization, in which the operating systems run within separate containers that may reside on a hypervisor.

[0010] In some cases, certain tasks requested by Process 112 can benefit from the use of Hardware Accelerators 130, which are designed to perform specific tasks. As with regard to Fig. As discussed in sections 4 to 8, the display unit 140 can perform various tasks to generate still images that are output through a display. The image signal processor (ISP) 150 can perform various tasks to process sensor data received from a camera. The audio unit 160 can perform various tasks to process input audio signals from a microphone and / or output audio signals for loudspeakers. The neural engine 170 can perform various tasks related to machine learning. The graphics unit 180 can perform various graphics processing tasks. However, as noted above, a challenge in allowing trusted processes 112A to use the hardware accelerators 130 is that they may represent a potential attack vector for untrusted processes 112B to gain access to the secure environment.

[0011] To prevent these types of circumvention in various embodiments, the hardware accelerators 130 are configured to implement isolated regions 132 of the secure environment, which extend the enforcement of one or more security criteria within the hardware accelerators 130. As described in more detail below, the accelerators 130 can implement the isolated regions 132 using a variety of techniques. For example, the accelerators 130 can prevent tasks assigned to untrusted processes 112B from accessing regions of memory 120 assigned to trusted processes 112A.In some embodiments, to implement this restriction, one or more hardware accelerators 130 store translations of virtual addresses to physical addresses for memory regions allocated to trusted processes 112A and for memory regions allocated to untrusted processes 112B, and prevent tasks assigned to untrusted processes 112B from accessing translations of virtual addresses to physical addresses for the memory regions allocated to trusted processes 112A. In some embodiments where an accelerator 130 does not access memory using virtual addresses, the accelerator 130 may maintain a table that lists the memory regions (e.g.,as defined by their physical addresses), which are assigned to the trusted processes 112A, and memory regions assigned to the untrusted processes 112B, and perform memory accesses according to the table. In various embodiments, the hardware accelerators 130 include additional switching logic to physically isolate the distributed data 122A assigned to the trusted processes 112A from the distributed data 122B assigned to the untrusted processes 112B. In some embodiments, this switching logic may include additional pipelines (or additional pipeline stages) configured to perform tasks assigned to the trusted processes 112A and to isolate these tasks from those assigned to the untrusted processes 112B.In some embodiments, this switching logic may include additional data buffers configured to store the trusted data 122A, which is distributed from memory 120 to perform tasks assigned to the trusted processes 112A. In some embodiments, one or more accelerators 130 include disconnect switches to enable or disable data paths providing data to the untrusted process 112, based on whether one or more security criteria are met.

[0012] As an exemplary use case, a processor 110 can execute a trusted process 112A to identify the utterance of a trigger word (e.g., "Hey Siri") in order to activate a voice assistant implemented by an untrusted process 112B. An isolated area 132C, implemented by the audio unit 160, can process input audio received from a microphone of the device 10 and provide the processed audio data to the trusted process 112A, which is permitted to access the audio data because process 112A is located in the secure environment.However, the audio unit 160 can initially prevent this audio data from leaving the isolated area 132C for the untrusted voice assistant process 112B, but negotiate one or more conditions under which audio data is allowed to leave the secure environment, such as 1) the trusted process 112A indicates that the trigger word has been detected, and 2) receiving an acknowledgment that the user is being notified about the microphone usage.

[0013] Further examples using the isolated areas 132 in the hardware accelerators 130 are given in relation to Fig. 4 to 8 are discussed in more detail. A software architecture used by the processors 110 to maintain the isolation of the trusted and untrusted processes 112 is next discussed in relation to Fig. 2 to 3C discussed.

[0014] Now with reference to Fig. Figure 2 shows a block diagram of a software architecture 200 of the device 10. As illustrated, the device 10 includes two operating systems: a first operating system 202 and a second operating system 212. In some examples, after being initially loaded into the device 10 by a boot program, an operating system manages processes and / or applications running on the device 10. In examples with multiple operating systems, each operating system manages its own processes and / or applications. It should be clear that more or fewer operating systems can be included in the device 10 and perform the techniques described herein.

[0015] As in Fig. As illustrated in Figure 2, the first operating system 202 includes a kernel 204, applications 206, and daemons 208. In some examples, the kernel 204 is a section of the first operating system 202 that is held in memory according to the first operating system 202 and / or supports interactions between software and hardware components according to the first operating system 202. For example, the kernel 204 can control hardware resources (e.g., I / O devices and / or memory) via device drivers, resolve conflicts between processes regarding such resources, and / or optimize the use of shared resources (e.g., CPU and / or cache usage, file systems, and / or network sockets). In some examples, the applications 206 include programs that are executed by a user to perform a specific task or service.In some examples, the daemons 208 include background processes that are run by the kernel 204 to perform a specific task or service for the applications 206, the kernel 204 and / or the first operating system 202.

[0016] As in Fig. As illustrated in Figure 2, the second operating system 212 includes a microkernel 214, services 216, drivers 218, and applications 220. Similar to the kernel 204, in some examples, the microkernel 214 is a section of the second operating system 212 that is held in memory in accordance with the second operating system 212 and / or supports interactions between software and hardware components in accordance with the first operating system 202. In some examples, the microkernel 214 provides less functionality compared to the kernel 204. For example, the microkernel 214 may support process scheduling, while other services, such as memory management and drivers for interacting with hardware, may be supported by other processes (e.g., the services 216 and the drivers 218).In some examples, the applications include 220 programs that are run through the second operating system 212 to perform a specific task or service.

[0017] As mentioned above, each of the two operating systems includes a kernel (e.g., kernel 204 for the first operating system 202 and microkernel 214 for the second operating system 212). It should be clear that the use of the terms "kernel" and "microkernel" is for illustrative purposes only, and one or both could be a different type of kernel in some of the examples described herein. For instance, microkernel 214 and / or kernel 204 could be a monolithic kernel, a microkernel, a hybrid kernel, a nanokernel, or an exokernel.

[0018] In some examples, the two operating systems of the device 10 described above operate at least partially independently of each other, although both use an overlapping portion of the device 10's resources (e.g., one or more processors, memory, I / O devices, and / or I / O interfaces). In some examples, the first operating system 202 operates in a normal execution mode, including the execution of one or more applications (e.g., applications 206) installed and / or stored on the device 10. In some examples, the one or more applications are unable to communicate directly with the second operating system 212 (e.g.,and / or a component of the second operating system 212), and instead communicate with the second operating system 212 via one or more system process(s) of the first operating system 202, such as the kernel 204 of the first operating system 202 and / or a daemon of the daemons 208 of the first operating system 202.

[0019] In some examples, the two operating systems of the device 10 described above are separated and / or isolated from each other by an isolation manager. The isolation manager manages interactions between the two operating systems 202 and the second operating system 212. For example, the isolation manager can provide memory sections and / or access to processors for one operating system during execution. As another example, the isolation manager can provide an interface so that the two operating systems can communicate with each other. In some examples, the isolation manager identifies what is running in a protected mode and what is running in a regular execution mode and provides access to specific resources based on which mode is currently in use. As in Fig. Figure 2 illustrates an example of the isolation manager as a secure page table monitor (SPTM) 210. In some examples, the SPTM 210 communicates with the first operating system 202 (e.g., via the kernel 204 and / or another process of the first operating system 202) and the second operating system 212 (e.g., via the microkernel 214 and / or another process of the second operating system 212). In such examples, the SPTM 210 can be used by the first operating system 202 and / or the second operating system 212 to receive identification of (e.g., allocation of) and / or access to memory of the device 10. It should be clear that other types of components can be used to provide separation and / or isolation for the two operating systems.

[0020] Fig. 3A to 3C are block diagrams that illustrate a data flow between the first sensor 308 and / or the second sensor 310 and an application 320 of a device 10 according to some examples. Each of the block diagrams of Fig. 3A to 3C is separated into three domains: a microkernel domain 302 (corresponding to the secure environment / isolated area domain), a kernel domain 304, and a user domain 306. In some examples, the microkernel domain 302 corresponds to operations that are managed and / or performed in a protected mode, as defined by the second operating system 212 in Fig. 2 are performed. In some examples, the kernel domain corresponds to 304 operations, which are performed by a system component of the first operating system in 202. Fig. 2 managed and / or performed (e.g., by a kernel process 204 and / or a daemon of daemons 208). In some examples, the user domain 306 corresponds to operations performed by an application (e.g., a non-system process) of the first operating system 202 in Fig. 2 are managed and / or executed (e.g., an application of application 206). In some examples, the kernel domain 304 and / or the user domain 306 operate in a regular execution mode, in contrast to the guarded mode described above with respect to the microkernel domain 302.

[0021] Now with reference to Fig. Figure 3A illustrates a block diagram of a flow 300 of sensor data from one or more sensor(s) (e.g. the first sensor 308 and / or the second sensor 310) through a filter layer 314 and a daemon 318 to an application 320.

[0022] As mentioned above, in some examples, the first sensor 308 and / or the second sensor 310 include a microphone, a touch-sensitive surface, a camera, a heart rate monitor, a pedometer, a depth sensor, a motion sensor, a magnetometer, and / or a gyroscope. For example, the first sensor 308 might be a camera, and the application 320 might be a photography application that requests an image from the first sensor 308 via the daemon 318 and the filter layer 314. In such an example, the daemon 318 and / or the filter layer 314 can determine whether the indicator 316 is turned on (e.g., active) before allowing the request to be fulfilled (and in some examples, the request would not be fulfilled if the indicator 316 is not turned on).

[0023] In Fig. 3A includes the flow where application 320 sends a request for sensor information (e.g., sensor data and / or metadata corresponding to sensor data) to daemon 318. In some examples, after receiving the request, daemon 318 determines the current context of device 10 (e.g., the location of device 10, what an output device communicating with device 10 is currently outputting, whether an output device communicating with device 10 is currently outputting, what an input device communicating with device 10 is currently detecting, whether an input device communicating with device 10 is currently detecting, which processes and / or applications are currently running on device 10, and whether a process and / or application is currently running on device 10).whether a process and / or an application is currently a background or foreground process on the device 10, whether the device 10 is currently communicating with another device and / or a specific device that is different from the device 10, what sensor data a sensor in communication with the output device 10 is currently acquiring, whether a specific sensor in communication with the output device 10 is currently acquiring sensor data, and / or a current time of day) and determines, based on the current context, which type of data to request and / or whether sensor information is to be requested from the filter layer 314 (e.g., different contexts may cause different types of data to be requested, or different types of data may causethat Daemon 318 either sends a request for sensor information to filter layer 314 or does not. For example, Daemon 318 can determine whether indicator 316 is turned on (e.g., active) (e.g., the current context of device 10), and in response to determining that indicator 316 is turned on, send a request for sensor information to filter layer 314. In some examples, the request for sensor information includes a specification of a sensor, a sensor type, and / or a type of sensor information requested. For example, the request for sensor information can include a request for an image from a camera. In other examples, Daemon 318 sends the request for sensor information to filter layer 314 without determining the current context of device 10.where such provisions are left to filter layer 314.

[0024] In Fig. 3A The first sensor 308 and the second sensor 310 (e.g., before, after, and / or as a result of a request for sensor information) acquire sensor data and send the sensor data to one or more secure services, drivers, or applications (e.g., secure drivers 312A to 312G). As described in Fig. As illustrated in Figure 3A, the first sensor 308 sends sensor data to the safe drivers 312A to 312D, and the second sensor 310 sends sensor data to the safe drivers 312A and 312E to 312G. It should be noted that the safe driver 312A receives sensor data from both the first sensor 308 and the second sensor 310, while the safe driver 312B receives sensor data from the first sensor 308 but not from the second sensor 310. It should be clear that one or more safe drivers can receive sensor data from more or fewer sensors, including from just a single sensor, or from a different sensor than shown. Fig. 3A illustrates or of three or more sensors. It should also be clear that a secure driver can correspond to any component of the second operating system 312, such as an application of applications 320, a driver of drivers 218, a service of services 216 and / or a component of the microkernel 214.

[0025] In some examples, sensor data received by a safe driver is acquired at different times and / or simultaneously. For example, the first sensor 308 can acquire initial sensor data and send it to the safe driver 312A, while the second sensor 310, after acquiring the initial sensor data, can acquire a second sensor data and send it to the safe driver 312A.

[0026] In some examples, a safe driver (e.g., the 312A to 312G safe drivers) receives sensor data and performs one or more operations, determinations, and / or calculations using that sensor data. For example, the safe driver can determine whether sensor data exceeds a threshold (e.g., a predefined threshold stored and / or configured for the safe driver, such as an amount of light, an amount of sound, a specific person in an image, a number of people in an image, a number of heartbeats, and / or whether an irregular heartbeat is present) and output a positive or negative response (sometimes referred to herein as metadata) based on whether the sensor data exceeded the threshold. In such an example, the positive or negative response can be sent to other safe drivers (e.g.,the safe drivers 312B to 312D and / or the safe drivers 312E to 312G) and / or are sent to the filter layer 314.

[0027] In some examples, one or more operations, determinations, and / or computations are provided to and / or specified for the safe driver (e.g., by a developer and / or a process (e.g., running in microkernel domain 302) associated with the safe driver) before or after the safe driver initiates execution. For example, the safe driver may include an interface description language (IDL) that defines how a component (e.g., daemon 318 and / or application 320) is able to interact with the safe driver through filter layer 314. In some examples, the IDL for the safe driver defines a message and / or request used to interact with the safe driver.For example, the IDL can define that a component can request whether the sensor data exceeds the threshold, thereby restricting interactions with the safe driver to whether the sensor data exceeds the threshold and preventing other types of interactions. In some examples, an IDL for a safe driver can include an inter-process communication (IPC) address for the safe driver, so that communications to the safe driver use IPC communication.

[0028] As mentioned above, the secure driver can output data (e.g., sometimes referred to as metadata, corresponding to sensor data) and send it to another secure driver and / or to filter layer 314. In some examples, the other secure driver receives metadata from the secure driver and also sensor data from a sensor (e.g., the first sensor 308 and / or the second sensor 310) and performs one or more operations, determinations, and / or calculations based on the metadata and sensor data to output its own metadata, which is then sent to another secure driver and / or to filter layer 314.

[0029] In Fig. 3A receives filter layer 314 metadata from one or more secure driver(s) and sensor data from one or more sensor(s) (e.g., collectively referred to as sensor information) and determines what to output outside filter layer 314. In some examples, filter layer 314 runs within microkernel domain 302 (e.g., as a service of services 216, an application of applications 220, or as part of microkernel 214), within kernel domain 304 (e.g., as a daemon of daemons 208, as part of kernel 204, or as a system process of kernel 204), or outside of microkernel domain 302 and kernel domain 304. In some examples, filter layer 314 acts as a gateway for sensor information from microkernel domain 302 to kernel domain 304. In some examples, filter layer 314 restricts access to sensor information from a component in kernel domain 304 and / or user domain 306.For example, the filter layer 314 of a camera can allow it to function as a light sensor and / or motion sensor without providing images to the daemon 318 by receiving metadata from a secure driver indicating whether light or motion is present, and then providing the metadata to the daemon 318 instead of an image (e.g., the images and / or any image).

[0030] In some examples, filter layer 314 determines the current context of device 10 and, based on this context, determines what should be output to daemon 318. For example, filter layer 314 can determine whether indicator 316 is turned on (e.g., active) and, in response to determining that indicator 316 is turned on, output sensor data (e.g., as in Fig. 3A illustrates) instead of metadata that corresponds to the sensor data (e.g., as in Fig. Figure 3B illustrates and is discussed in more detail below), in response to the sensor information requirement discussed above, Daemon 318 outputs the following information. In some examples, Daemon 318 determines a current context of the device 10, similar to or identical to what is described above regarding filter layer 314, and determines what to output to application 320 based on that current context. In other examples, Daemon 318 outputs what it has received from filter layer 314 to application 320 without any determination of the current context.

[0031] As described above, filter layer 314 and / or daemon 318 can determine the current context of device 10. In some examples, as part of determining the current context of device 10, filter layer 314 and / or daemon 318 send a request to change a state of device 10 to ensure that its current context meets a set of criteria required to process a request. For example, if indicator 316 must be turned on (e.g., the set of criteria includes a criterion that is met when indicator 316 is turned on), filter layer 314 and / or daemon 318 can send a request to ensure that indicator 316 is turned on before proceeding with a current request.Accordingly, if the request to trigger indicator 316 is successful, the current request can be processed without changing the type of data provided in response.

[0032] In some examples, the sensor data provided to Daemon 318 and Application 320 has a different resolution than that acquired by a sensor (e.g., the first sensor 308 and / or the second sensor 310). In such examples, Filter Layer 314 determines, based on the current context of Device 10, which resolution of sensor data should be sent to Daemon 318 and / or Application 320. In some examples, Filter Layer 314 sends sensor data at that resolution to Daemon 318. For example, the sensor may provide images at a rate of 1 per millisecond, while Filter Layer 314 and / or Daemon 318 may provide images at a rate of 1 per second (e.g., a lower resolution than 1 per millisecond).

[0033] Now with reference to Fig. Figure 3B illustrates a block diagram of a flow of sensor data from one or more sensor(s) (e.g. the first sensor 308 and / or the second sensor 310) to one or more secure driver(s) 312A to 312G and metadata from the one or more secure driver(s) to a filter layer 314, a daemon 318 and then to an application 320.

[0034] As above in relation to Fig. 3A described, illustrated Fig. 3B, that the first sensor 308 and the second sensor 310 send sensor data to the safe drivers 312A to 312G and to the filter layer 314, and that the safe drivers 312A to 312G send metadata to the filter layer 314. However, instead of the filter layer 314 sending sensor data to the daemon 318, as in Fig. As illustrated in 3A, filter layer 314 sends metadata to daemon 318.

[0035] In some examples, the metadata is sent from daemon 318 to daemon 318 in response to a request for metadata (e.g., daemon 318 either received a request for metadata from application 320, or daemon 318 determined that a current context of device 10 requires metadata to be provided instead of sensor data). In other examples, the metadata is sent from daemon 318 to daemon 318 in response to a request for sensor information (e.g., sensor data and / or metadata) (e.g., daemon 318 either received a request for sensor information, sensor data, or metadata from application 320, or daemon 318 determined that a current context of device 10 requires filter layer 314 to determine what to send to daemon 318 in response). As in Fig. Figure 3B illustrates that indicator 316 is switched off (e.g., inactive) and not switched on (as in, for example, Fig. Figure 3A illustrates this). In some examples, the fact that indicator 316 is turned off causes metadata, rather than sensor data, to be sent to daemon 318.

[0036] Although described above as metadata, it should be clear that metadata can have different resolutions (e.g., include different amounts of data and / or specificity). In some examples, filter layer 314 determines, based on the current context of device 10, which resolution of metadata should be sent to daemon 318 and / or application 320, and sends metadata of that resolution to daemon 318. For example, filter layer 314 may receive an initial indication that a specific person is captured in an image and a second indication that a person is present in an environment (e.g., the specific person is identified in the first indication and not in the second). In such an example, filter layer 314 may send daemon 318 either the first indication (e.g.,a higher resolution than the second specification, since the specific person is identified) or the second specification (e.g. a lower resolution than the first specification, since the specific person is not identified) depending on the current context of the device 10 and / or which application requests such information.

[0037] Now with reference to Fig. 3C illustrates a block diagram of the flow of sensor data from one or more sensor(s) (e.g. the first sensor 308 and / or the second sensor 310) to one or more secure driver(s) 312A to 312G and a filter layer 314, of metadata from the one or more secure driver(s) to the filter layer and neither the metadata nor the sensor data to a daemon 318 and an application 320.

[0038] As above in relation to Fig. 3A to 3B described, illustrated Fig. 3C, that the first sensor 308 and the second sensor 310 send sensor data to the safe drivers 312A to 312G and to the filter layer 314, and that the safe drivers 312A to 312G send metadata to the filter layer 314. However, instead of the filter layer 314 sending sensor data (such as in Fig. 3A illustrates) or metadata (such as in Fig. (as illustrated in 3B) sends to Daemon 318, Filter Layer 314 sends data to Daemon 318 indicating that sensor data and metadata cannot be sent based on a current context of Device 10.

[0039] In some examples, the data is sent from daemon 318 to daemon 318 in response to a request for sensor information (e.g., sensor data and / or metadata). This applies, for example, daemon 318 either receives a request for sensor information, sensor data, or sensor metadata from application 320, or daemon 318 determines that a current context of device 10 requires filter layer 314 to determine what to send to daemon 318 in response. As in Fig. As illustrated in 3C, indicator 316 is turned off (e.g., inactive). In some examples, the fact that indicator 316 is turned off causes data, rather than sensor data or metadata, to be sent to daemon 318. It should be clear that a different context can cause filter layer 314 to send sensor data of varying resolutions, metadata of varying resolutions, or neither to daemon 318.

[0040] It should be clear that, while Fig. Sections 3A to 3C illustrate a single application (e.g., application 320) that interacts with filter layer 314 via daemon 318. Different applications (e.g., of the same type or of different types) can communicate with filter layer 314 via daemon 318 (or another daemon) using the same operations with the same or different criteria as described above for application 320. For example, filter layer 314 and / or daemon 318 can have different criteria for different applications and / or different types of applications when determining how to respond to a request.

[0041] Several examples of hardware accelerators 130 that use isolated areas 132 to extend the secure environment provided by the Fig. 2 and Fig. The software architecture described in section 3 is provided.

[0042] Now with reference to Fig. Figure 4 shows a block diagram of the display unit 140. As noted above, the display unit 140 includes switching logic configured to render individual frames for display on a screen of the device 10, as shown below with reference to Fig. 12 is discussed. In the illustrated embodiment, the display unit 140 includes a display pipeline 410, which includes a safe mix 412 and a safe extractor 416. The display unit further includes a safe direct memory access (DMA) engine 420. In other embodiments, the display unit 140 may be implemented differently than shown—for example, the DMA engine 420 may be considered part of the pipeline 410.

[0043] The display pipeline 410 is a switching logic configured to process input data 402 to generate output single frames 404. Accordingly, the pipeline 410 can be configured to perform various operations, such as panel compensation, swizzling, dithering, cropping, timing, DisplayPort™ transmission, etc. In the illustrated embodiment, the display unit 140 is configured to implement a safe area 132A to extend the enforcement of one or more safe environment safety criteria with respect to the pipeline 410 by using a safe mix 412, a safe extractor 416, and a safe DMA 420.Here, elements 412 to 420 serve to physically isolate the distributed trusted data 122A by providing a separate data path controlled from the secure environment by one or more processes 112A. This allows the pipeline 410 to perform tasks assigned to the untrusted processes 112B and the trusted processes 112A. As shown, the isolated area 132A provides a way for a trusted process 112 to have pixel data 414 combined with input pixel data 402 provided by an untrusted process 112B.

[0044] The secure mix 412 is an additional pipeline stage that includes switching logic configured to insert pixel data 414 from a trusted process 112A into an output frame 404 before the output frame 404 is displayed. For example, the inserted pixel data 414 may appear as one or more colored dots in the upper right corner of the frame 404. As shown, the inserted pixel data 414 can be used to transmit an indicator 316 to an active component of the device 10, such as a sensor configured to collect sensitive data (e.g., health, location, etc.) about a user, a camera, a microphone, hardware interfaces, etc.The inserted pixel data 414 can also be used to transmit indicators 316 associated with specific software, such as notifying a user that screen sharing is occurring, credentials are being accessed, payment information is being sent to a merchant—or even to transmit notifications associated with non-sensitive information. In some embodiments, the pixel data 414 received by the secure mixture 412 is generated by a trusted process 112A responsible for notifying a user. In other embodiments, the secure mixture 412 can itself generate the pixel data 414 in response to a request (or other indication) from the secure environment.The safe mix 412 can also perform other tasks, such as adding a ring around inserted pixel data 414 to prevent it from being obscured when inserted into a single image 404 that has the same color background.

[0045] The secure extractor 416 is an additional pipeline stage that includes switching logic configured to extract the pixel data 414 from where it was previously inserted into an output single frame 404, to confirm that it is still present in the single frame 404 directed to the display. The secure extractor 416 can also check other details, such as performing a CRC check, verifying that the display is active and communicating with the pipeline 410, etc. In the illustrated embodiment, the secure extractor 416 provides the extracted pixel data to a trusted process 112A for analysis (such as a process 112A associated with the filter layer 314) – although in other embodiments, the extractor 416 can analyze the data 414 locally and transmit a result.If the analysis determines that the pixel data 414, corresponding to indicator 316, should remain embedded in the output image 404, the usage associated with indicator 316 may be allowed to continue. Otherwise, corrective actions may be taken, as described in [reference to relevant section]. Fig. 5 and Fig. 6 is discussed. In various embodiments, the safe extractor 416 is configured to provide periodically extracted pixel data 414, so that the presence of the inserted pixel data 414 can be continuously confirmed.

[0046] The safe DMA engine 420 is switching logic configured to transmit pixel data 414 between the isolated area 132A and the memory regions 120 allocated to the trusted processes 112A. In the illustrated embodiment, the safe DMA engine 420 is a second DMA engine, controlled separately from the primary DMA engine, configured to retrieve input data 402 from memory 120 for the display pipeline 410. The safe DMA engine 420 may also be inaccessible to components outside the safe environment and may only process requests originating from the safe environment or an isolated area 132.By providing this separate data path, the secure DMA engine 420 can ensure that components of the isolated area 132A do not process data 112B received from the untrusted processes 112B, in order to make it more difficult for a malicious process 112B to disrupt tasks performed by the isolated area 132A.

[0047] Several features described above with respect to the display unit 140 can also be used by other user interface switching logic, such as audio pipeline switching logic in the audio unit 160 to insert an audible indicator into an audio signal, a haptic feedback engine to insert a specific haptic indicator into haptic feedback data, and so on. In some embodiments, the display unit 140 can also implement techniques used by other accelerators 130, such as IOMMUs. The hardware accelerators 130, which can control access to sensor data based on the extracted pixel data 414, will now be described next with respect to Fig. 5 and Fig. 6 described.

[0048] Now with reference to Fig. Figure 5 shows a block diagram of the image signal processor (ISP) 150. As noted above, the ISP 150 includes sensor processor switching logic configured to process sensor data received from a camera. In the illustrated embodiment, the ISP 150 includes sensor pipelines 510, an input-output memory management unit (IOMMU) 520, a disconnect switch 530, and a safety processor 540. In other embodiments, the ISP 150 may be implemented differently than shown; techniques described in relation to the ISP 150 may be implemented by other hardware accelerators 130; the ISP 150 may implement techniques other than those described in connection with other hardware accelerators 130, and so on.

[0049] The sensor pipelines 510 include switching logic configured to perform various image processing operations, such as sensor compensation, color space encoding, scaling, rotation, format encoding, etc. In the illustrated embodiment, the ISP 150 implements an isolated area 132B to extend the enforcement of one or more safety criteria of the safe environment by using a safe pipeline 510A, the IOMMU 520, the disconnect switch 530, and the safe processor 540.In this process, the ISP 150 can securely provide processed camera sensor data 502 to the trusted processes 112A (or other trusted consumers) and negotiate one or more conditions under which the untrusted processes 112B (or other untrusted consumers) are allowed to receive processed sensor data 502 - such as confirmation that the indicator 316 is currently being provided to a user via the pixel data 414.

[0050] The secure pipeline 510A is an additional pipeline that provides data isolation for the camera sensor data 502 for the trusted processes 112, while an insecure pipeline 510B processes the data 502 for the unsecured processes 112. In some cases, it may be desirable to perform certain tasks without burdening the user with additional indicators 316. For example, a trusted process 112A can continuously analyze the camera sensor data 502 processed by pipeline 510A to determine whether a user is paying attention to the device 10, in order to potentially implement various energy-saving techniques, such as dimming the display when the user is not paying attention to the device, etc. A trusted process 112A or trusted hardware can also be used to perform biometric authentication of a user using facial recognition, iris recognition, etc.Using a secure pipeline 510A separate from pipeline 510B can make it more difficult for a malicious process 112B to circumvent the security criteria enforced by the isolated area 132B. This can also prevent a malicious process 112B from blocking access to the camera sensor data 502 used by the trusted processes 112A. In some embodiments, using pipeline 510A can also allow the ISP 150 to provide the trusted processes 112A with greater capabilities that may be less desirable for the untrusted processes 112B, such as higher resolutions, frame rates, etc.To avoid mixing the tasks of trusted and untrusted processes 112, the pipelines 510 can be addressed separately by the processes 112, so that only processes 112A within the secure environment (or other trusted entities) can address resources assigned to the secure pipeline 510A. The ISP 150 can also receive other information to distinguish between trusted and untrusted entities, such as information received from the fabric 102 about the sources of requests and address translations 522, which is discussed below. In the illustrated embodiment, the pipelines 610 also use separate DMA engines 612 to issue memory requests to the memory 120.As shown, the direct DMA engine 512A is configured to write the sensor data 502 processed by pipeline 510A to a section of memory 120 accessible to the trusted processes 112A (or regions accessible to other trusted consumers); the DMA engine 512B is configured to write the sensor data 502 processed by pipeline 510B to another section of memory accessible to the untrusted processes 112B (or regions accessible to other untrusted consumers).

[0051] The IOMMU 520 is a switching logic unit configured to forward memory requests from the DMAs 512, via Fabric 102, to memory 120. As part of this communication, the IOMMU 520 translates virtual addresses specified in the memory requests into their corresponding physical addresses known to memory 120. In the illustrated embodiment, the IOMMU 520 prevents tasks assigned to the untrusted processes 112B from accessing memory regions assigned to the trusted processes 112A by storing separate sets of address translations 522A and 522B for the trusted processes 112A and the untrusted processes 112B, respectively.Accordingly, when the IOMMU 520 receives a memory request DMA 512B for an untrusted process 112B, it accesses its untrusted address translations 522B and, if a corresponding translation 522B is stored, translates the virtual address specified in the request to its corresponding physical address for communication with memory 120. However, if no corresponding translation 522B is stored, the IOMMU 520 is unable to perform the translation—thus preventing tasks assigned to the untrusted processes 112 from accessing unauthorized memory regions, such as those assigned to the trusted processes 112A.Tasks assigned to trusted processes 112A and handled by the secure pipeline 510A can also be prevented from accessing regions assigned to untrusted processes 112B if no corresponding translation is stored in the set of trusted address translations 522A. The IOMMU 520 can determine, based on the specific DMA 512A or 512B that issues the memory request, which set of translations 522A or 522B to access for a given memory request. In some embodiments, the translations 522A are provided by the SPTM 210 discussed above.

[0052] The disconnect switch 530 is a switching logic device configured to control whether sensor data 502 is allowed to flow to sensor pipeline 510B and thus to an untrusted process 112B. In response to one or more conditions for providing the data 502 being met (such as the indicator 316 provided via display unit 140), the switch 530 allows the data 502 to flow to pipeline 510A. However, if one or more of the conditions are violated, the switch 530 is configured to interrupt the data path through which the data 502 is provided to pipeline 510A. In some embodiments, the disconnect switch 530 is configured as a dead man's switch, which remains activated while periodic confirmations that the criteria are met are received, but which interrupts the provision of the data 502 to the pipeline 510A in response to a lack of confirmation.In some embodiments, the disconnect switch 530 is controlled by the safe processor 540 based on an indicator confirmation 544.

[0053] The secure processor 540 is a processor configured to manage various operations of the ISP 150. To allow the secure processor 540 to be controlled from within the secure environment by one or more trusted processes 112A, the secure processor 540 may include one or more configuration registers configured to store configuration information 542 that controls the operation of the camera sensor and is addressable only by processes 112A within the secure environment. This configuration information 542 can be used to control specific camera settings, control the behavior of the pipeline 510, and so on.In the illustrated embodiment, when the camera sensor data 502 is accessed by an untrusted process 112B, the secure processor 540 receives an indicator acknowledgment 544 confirming whether the indicator 316 is displayed to the user (or more generally, that one or more criteria for providing data 502 have been met). As noted above, this acknowledgment 544 can be provided by a trusted process 112A, which analyzes the extracted pixels 414 from the display unit 140. The secure processor 540 can also be responsible for providing an indication that the camera sensor is active to a corresponding process 112A, which is executable to generate the indicator 316.In some embodiments, the secure processor 540 periodically receives an indicator acknowledgment 544 while the sensor is active, as a heartbeat signal indicating that one or more conditions for providing access to the camera sensor data 502 have been met. Upon determining that the heartbeat signal is no longer being received, the sensor processor 540 can take one or more corrective actions to stop providing the sensor data 502 to an untrusted process 112B. In the illustrated embodiment, the secure processor 540 is coupled to a sensor power management unit 550 configured to supply power to the camera sensor. Upon determining that one or more conditions for providing access to the sensor data 502 have been violated, the secure processor 540 can instruct the PMU 550 to turn off the sensor.The processor 540 can also instruct the switch 530 to interrupt the data path that provides the data 502.

[0054] Although the description of the ISP 150 is presented in the context of a camera, this description is also applicable to other sensor types, such as those mentioned above, and a microphone, which will be discussed next.

[0055] Now with reference to Fig. Figure 6 shows a block diagram of an audio unit 160. As noted above, the audio unit 160 includes sensor processor switching logic configured to process audio signals received from one or more microphone sensors of the device 10. In some embodiments, the audio unit 160 also includes switching logic (not shown) for driving one or more loudspeakers. In the illustrated embodiment, the audio unit 160 includes the sensor pipelines 610, an IOMMU 620, and a disconnect switch 630. In other embodiments, the audio unit 160 may be implemented differently than shown; techniques described with respect to the audio unit 160 may be implemented by other hardware accelerators 130; the audio unit 160 may implement techniques other than those discussed in connection with other hardware accelerators 130, etc.

[0056] Sensor pipelines 610A and 610B include switching logic configured to perform audio signal processing, such as amplifiers, analog-to-digital converters (ADCs), filters, digital signal processors (DSPs), audio encoders, DMA engines 612, etc. In the illustrated embodiments, the audio unit 160 implements an isolated area 132C to extend the enforcement of one or more safety criteria of the safe environment by using the safe pipeline 610A, the IOMMU 620, and the disconnect switch 630. Similar to pipeline 510A, the safe pipeline 610A is an additional pipeline that provides data isolation for the audio sensor data 602 for the trusted processes 112 and can be used independently of the untrusted pipeline 510B, which processes the data 602 for the untrusted processes 112B.To maintain this separation, pipeline 610A also includes a separate DMA 612A to ensure data separation for storage requests issued from the secure pipeline 610A from storage requests issued from the secure pipeline 610B via DMA 612B.The IOMMU 620 is similarly configured to serve these memory requirements by storing a first set of memory address translations 622A (and thus memory addresses) designated as accessible to the safe pipeline 610A, and a second set of memory address translations 622B designated as accessible to the unsafe pipeline 610B, preventing the safe pipeline 610A from accessing the memory address translations 622A (and thus memory addresses associated with these translations 622A) outside the first set, and preventing the unsafe pipeline 610B from accessing the memory address translations 622B outside the second set.The disconnect switch 630 is similarly configured to enable or disable the provision of audio sensor data 602 to an untrusted process 112B via sensor pipeline 610B in response to one or more security criteria being met.

[0057] Now with reference to Fig. Figure 7 shows a block diagram of a neural engine 170. As noted above, the neural engine 170 includes switching logic configured to perform machine learning operations, such as those associated with neural networks. As shown, the neural engine 170 includes a neural engine core 710, an IOMMU 720, and several context queues 730. In other embodiments, the neural engine 170 may be implemented differently than shown; techniques described in relation to the neural engine 170 may be implemented by other hardware accelerators 130; the neural engine 170 may implement techniques other than those described in connection with other hardware accelerators 130, and so on.

[0058] The core of the Neural Engine 710 includes switching logic configured to perform various neural network operations, such as those associated with matrix multiplication, activation function application, backpropagation computation, and various other tensor operations. As demonstrated, the core of the Neural Engine 710 can be used to perform sensitive tasks assigned by trusted processes 112A, which may include user authentication, speech recognition for activating a voice assistant, attention detection, and so on.To ensure the separation of tasks assigned by the untrusted processes 112A and tasks assigned by the untrusted processes 112B, the neural engine 170 implements an isolated area 132D using the IOMMU 720 and one or more secure queues 730A.

[0059] In various embodiments, separate context queues 730 are used to preserve the state for separate contexts associated with processes 112A and 112B. When switching between task execution, the core of the neural engine 710 can implement a context switch, where the state from one task execution is offloaded to a queue 730, and the state for another task execution is loaded from a queue 730 into the core 710. In the illustrated embodiment, a separate, secure context queue 730A is used to physically isolate the state data 122A belonging to the trusted processes 112A.Since this separate secure context queue 730A serves as an additional data buffer for storing the distributed data 122A associated with the trusted processes 112A, and does not store any distributed data 122B associated with the untrusted processes 112B, the distributed data 122A can be protected from the untrusted processes 112B. Furthermore, the IOMMU 720 can restrict access to queues 730 by processes 112A and 112B using the untrusted and trusted address translations 722, as discussed above in conjunction with IOMMUs 520 and 620, and limit the memory requirements issued by the contexts associated with queues 730.

[0060] Now with reference to Fig. Figure 8 shows a block diagram of the Graphics Unit 180. As noted above, the Graphics Unit 180 includes switching logic configured to render graphical content—or to perform other forms of highly parallelized tasks, as described in relation to Fig. 12 is discussed. As shown, the graphics unit 180 can include one or more graphics processor unit core(s) (GPU core(s), GPU = graphics processor unit) 810 and an address resolution table 820. In other embodiments, the graphics unit 180 can be implemented differently than shown; techniques described with respect to the graphics unit 180 can be implemented by other hardware accelerators 130; the graphics unit 180 can implement techniques other than those described in connection with other hardware accelerators 130, etc.

[0061] As with other accelerators 130, the GPU cores 810 can be configured to perform tasks assigned by trusted processes 112A and tasks assigned by untrusted processes 112B. To ensure data separation between the trusted process data 122A and the untrusted process data 122B during the execution of these tasks, the graphics unit 180, in the illustrated embodiment, implements an isolated area 132E using an address resolution table 820. In some embodiments, the GPU cores 810 (or other components of the graphics unit 180, such as the internal CPU management cores 810) issue memory requests that specify the physical addresses of the memory 120 (as opposed to virtual addresses corresponding to those physical addresses).Since no virtual address translation is performed, the graphics unit 180 cannot use an IOMMU, as discussed above in conjunction with other accelerators 130. Instead, an address resolution table 820 maintains the trusted and untrusted address mappings 820, which identify the memory regions of memory 120 allocated to the trusted processes 112A and memory regions of memory 120 allocated to the untrusted processes 112B. In some embodiments, these mappings 822 are provided by the SPTM 210 discussed above and can specify the physical addresses accessible for a task assigned to a given process 112A or 112B.Accordingly, when GPU cores 810 are performing tasks assigned to an untrusted process 112B, they can issue memory requests specifying physical addresses for table 820. If a corresponding mapping 822 exists in table 820 for this process 112B, the memory requests can be allowed to proceed to memory 120 via fabric 102. Otherwise, these memory requests can be blocked by table 820. In some embodiments, the isolated area 132E also uses separate DMA engines to issue memory requests with respect to data 122A and 122B, as discussed above.

[0062] Now with reference to Fig. Figure 9A shows a flowchart of Method 900. Method 900 is an embodiment of a method carried out by a data processing device, such as Device 10. In many cases, carrying out Method 900 can significantly improve the security of the data processing device. In some embodiments, Method 900 may be implemented differently than shown.

[0063] In step 910, one or more processor(s) (e.g., the processors 110) of the data processing device jointly execute trusted processes (e.g., the trusted processes 112A) and untrusted processes (e.g., the untrusted processes 112B) in an isolated manner, which includes implementing a secure environment in which a set of security criteria is enforced for data of the trusted processes (e.g., the trusted process data 122A).

[0064] In step 920, several heterogeneous hardware accelerators (e.g., the hardware accelerators 130) implement isolated areas (e.g., the isolated areas 132) of the secure environment that extend the enforcement of one or more of the set of security criteria within the hardware accelerators for data distributed to the hardware accelerators for the execution of tasks assigned to the trusted processes. In various embodiments, to extend the enforcement of the one or more security criteria, one or more of the hardware accelerators prevent tasks assigned to the untrusted processes from accessing memory regions (e.g., of memory 120) allocated to the trusted processes.In some embodiments, to limit tasks, the one or more hardware accelerators store translations of virtual addresses to physical addresses (e.g., translations 522, 622, and 722) for the memory regions allocated to trusted processes and for memory regions allocated to untrusted processes, and prevent tasks assigned to untrusted processes from accessing the translations of virtual addresses to physical addresses for the memory regions allocated to trusted processes. In some embodiments, the one or more hardware accelerators include a graphics processing unit (GPU) (e.g., GPU 180) that stores a table (e.g.,the address resolution table (820) which identifies the memory regions allocated to trusted processes and the memory regions allocated to untrusted processes, and performs memory accesses according to the table.

[0065] In various embodiments, one or more of the hardware accelerators physically isolate the distributed data associated with trusted processes from the data associated with untrusted processes to enhance the enforcement of the one or more security criteria. In some embodiments, the one or more hardware accelerators include pipelines (e.g., pipelines 410, 510, and 610) that perform tasks associated with untrusted processes and tasks associated with trusted processes. In some embodiments, the pipelines include one or more additional pipeline stages that process the distributed data associated with trusted processes and do not process the distributed data associated with untrusted processes.In some embodiments, the one or more hardware accelerators include a display unit comprising a display pipeline that renders single frames for display on the data processing device. In some embodiments, the display pipeline includes a mixing pipeline stage (e.g., the safe mix 412) that mixes pixel data (e.g., the inserted pixel data 414) received from a trusted process into a single frame (e.g., the output single frame 404) rendered by the display pipeline, which includes pixel data from an untrusted process. In some embodiments, the display pipeline includes an extraction pipeline stage (e.g., the safe extractor 416) that extracts pixel data (e.g., the extracted pixel data 414) from a single frame rendered by the display pipeline and provides the extracted pixel data to a trusted process.In some embodiments, the one or more hardware accelerators include one or more additional data buffers that store distributed data associated with trusted processes and do not store distributed data associated with untrusted processes. In some embodiments, the one or more hardware accelerators include a neural engine (e.g., Neural Engine 170) that performs a set of neural network operations and includes the one or more data buffers (e.g., Secure Context Queue 730A) to store distributed data for tasks associated with trusted processes.

[0066] In various embodiments, one or more of the hardware accelerators, in order to extend the enforcement of one or more security criteria, prevent sensor data provided to trusted processes from being provided to untrusted processes. In some embodiments, the one or more hardware accelerators include an image signal processor (e.g., the ISP 150) that processes sensor data received from a camera sensor (e.g., the camera sensor data 502) and includes a disconnect switch (e.g., the disconnect switch 530) to enable or disable the provision of sensor data to an untrusted process in response to one or more security criteria being met.In some embodiments, the one or more criteria include an indication present in a single image displayed to a user to indicate that the camera sensor is currently in use. In some embodiments, the one or more hardware accelerators include an audio unit (e.g., the audio unit 160) that processes sensor data received from a microphone sensor (e.g., the sensor data 602) and includes a disconnect switch (e.g., the disconnect switch 630) to enable or disable the provision of the sensor data to an untrusted process in response to one or more security criteria being met.

[0067] Now with reference to Fig. Figure 9B shows a flowchart of Method 930. Method 930 is an embodiment of a method carried out by a data processing device, such as Device 10. In many cases, carrying out Method 930 can significantly improve the security of the data processing device. In some embodiments, Method 930 may be implemented differently than shown.

[0068] Procedure 930 begins in step 935 with the data processing device isolating jointly executed trusted processes (e.g., the trusted processors 112A) and untrusted processes (e.g., the untrusted processes 112B). In step 940, the data processing device distributes data to some of the multiple heterogeneous hardware accelerators to perform tasks requested by the processes. In step 945, the data processing device's heterogeneous hardware accelerators receive information about how the trusted and untrusted processes are isolated. In step 950, based on the information received, the heterogeneous hardware accelerators extend the isolation of the trusted and untrusted processes to jointly execute tasks that work on the distributed data.

[0069] Now with reference to Fig. Figure 9C shows a flowchart of Method 960. Method 960 is an embodiment of a method carried out by a data processing device, such as Device 10. In many cases, carrying out Method 960 can significantly improve the security of the data processing device. In some embodiments, Method 960 may be implemented differently than shown.

[0070] Procedure 960 begins in step 965 with one or more processors (e.g., processors 110) of the data processing device jointly executing trusted processes (e.g., trusted processes 112A) and untrusted processes (e.g., untrusted processes 112B) such that the trusted processes are isolated from the untrusted processes. In step 970, several heterogeneous hardware accelerators (e.g., accelerators 130) of the data processing device 10 perform tasks requested by the trusted processes. In step 975, the hardware accelerators negotiate conditions under which tasks requested by the untrusted processes are permitted to be executed.

[0071] Now with reference to Fig. Figure 10 shows a flowchart of a method 1000. The method 1000 is an embodiment of a method performed by a data processing device, such as the device 10, which includes a sensor and sensor-processor switching logic. In many cases, performing the method 1000 can significantly improve the security of the data processing device. In some embodiments, the method 1000 may be implemented differently than shown. In step 1005, the sensor-processor switching logic processes sensor data received from a sensor of the data processing device. In some embodiments, the sensor-processor switching logic is an image signal processor (e.g., the ISP 150) configured to process sensor data received from a camera. In some embodiments, the sensor-processor switching logic is an audio processor (e.g.,the audio unit 160), which is configured to process sensor data received from a microphone.

[0072] In step 1010, a sensor processor switching logic, in response to an initial indication that a first consumer is trusted, provides an initial data set of processed sensor data for the first consumer. In various embodiments, the initial indication identifies the first consumer as being in a safe environment where a set of safety criteria is enforced for the first data set; the second indication identifies the second consumer as being outside the safe environment. In some embodiments, the data processing device implements a safe execution environment of the safe environment, in which the first consumer is a first process (e.g., a trusted process 112A) running within the safe execution environment, and the second consumer is a second process (e.g., a trusted process 112A).the untrusted process 112B) is executed outside the safe execution environment. In some embodiments, the sensor processor switching logic includes one or more configuration registers (e.g., in the safe processor 540) that store configuration information (e.g., the configuration information 542) which controls the operation of the sensor and is addressable only by entities within the safe environment.

[0073] In step 1015, in response to a second indication that a second consumer is not trusted, the sensor-processor switching logic negotiates one or more conditions under which the second consumer is permitted to receive a second dataset of the processed sensor data. In some embodiments, a sensor-processor switching logic provides an indication that the sensor is active. In some embodiments, the one or more conditions include the sensor-processor switching logic receiving confirmation that a user will be notified that the sensor is active. In some embodiments, a display pipeline switching logic (e.g., the display pipeline 410) of the data processing device inserts pixel data (e.g., the pixel data 414) into a single image (e.g., a display) in response to the provided indication.the output single image 404), which is displayed on a screen to notify the user that the sensor is active. In some embodiments, in response to the determination to provide the second data set to the second consumer, the sensor processor switching logic periodically receives a heartbeat signal (e.g., the indicator confirmation 544) indicating that one or more conditions have been met, and stops providing the second data set in response to the determination that the heartbeat signal is no longer being received. In some embodiments, a switch (e.g., the disconnect switches 530 and 630) of the sensor processor switching logic interrupts a data path through which the second data set is provided to the second consumer in response to one or more conditions being violated. In some embodiments, the sensor processor switching logic (e.g.,The sensor PMU 550) shuts down the sensor in response to determining that one or more conditions have been violated. In some embodiments, a safe pipeline (e.g., the safe pipelines 510A and 610A) of the sensor processor switching logic processes sensor data to generate the first data set for the first consumer; an unsafe pipeline of the sensor processor switching logic processes sensor data to generate the second data set for the second consumer. In some embodiments, an input-output memory management unit (IOMMU) (e.g.,The IOMMUs 520, 620, and 720 of the data processing device store a first set of memory addresses designated as accessible to the secure pipeline and a second set of memory addresses designated as accessible to the insecure pipeline, preventing the secure pipeline from accessing memory addresses outside the first set and the insecure pipeline from accessing memory addresses outside the second set. In some embodiments, the memory addresses are stored as translations from virtual to physical addresses (e.g., translations 522, 622, and 722).In some embodiments, the sensor processor switching logic includes a first direct memory access (DMA) engine configured to write the first data set to a section of memory accessible to the first consumer, and a second DMA engine configured to write the second data set to a different section of memory accessible to the second consumer.

[0074] Now with reference to Fig. Figure 11A shows a flowchart of a method 1100. Method 1100 is an embodiment of a method performed by a data processing device, such as device 10, which includes a user interface and user interface pipeline switching logic. In many cases, performing method 1100 can significantly improve the security of the data processing device. In some embodiments, method 1100 may be implemented differently than shown.

[0075] In step 1105, the user interface pipeline switching logic (e.g., a pipeline in the display unit 140, the audio unit 160, etc.) of the data processing device processes a set of data (e.g., the input data 402) received from a first source (e.g., a trusted process 112A) to generate an output for the user interface of the data processing device. In some embodiments, the user interface pipeline switching logic is a display pipeline switching logic (e.g., the display pipeline 410) configured to generate single frames (e.g., the output single frame 404) for a display of the data processing device. In some embodiments, the user interface pipeline switching logic is an audio pipeline switching logic configured to generate an audio signal for a loudspeaker of the data processing device.In some embodiments, the user interface pipeline switching logic is a haptic pipeline switching logic configured to generate haptic feedback data for a haptic feedback engine of the data processing device.

[0076] In step 1110, the user interface pipeline switching logic receives a signal from a second source indicating that a component of the data processing device has been activated. In some embodiments, the component is a sensor configured to collect sensitive data about a user, a camera configured to collect image data, or a microphone configured to capture audio data. In some embodiments, the set of data is provided to the user interface pipeline switching logic via a first untrusted process (e.g., untrusted process 112B) corresponding to the first source; the signal is provided to the user interface pipeline switching logic via a second trusted process corresponding to the second source.In some embodiments, the user interface pipeline switching logic includes a first direct memory access (DMA) engine configured to retrieve the set of data from memory, and a second, different DMA engine (e.g., the DMA 420 engine) configured to retrieve the specification from memory.

[0077] In step 1115, the user interface pipeline switching logic inserts an indicator of the enabled component into the output before displaying it through the user interface. In some embodiments where the user interface pipeline switching logic is a display pipeline switching logic, the display pipeline switching logic includes a mixing pipeline stage (e.g., the safe mix 412) configured to insert pixel data (e.g., the inserted pixel data 414) as the indicator into a single frame rendered by the display pipeline switching logic, based on the received specification. In some embodiments where the user interface pipeline switching logic is audio pipeline switching logic, the audio pipeline switching logic includes a mixing pipeline stage configured to insert an audio indicator into the audio signal based on the received information.In some embodiments where the user interface is a haptic feedback engine, the haptic pipeline switching logic includes a mixing pipeline stage configured to insert a specific haptic indicator into the haptic feedback data based on the received input.

[0078] In various embodiments, Method 1100 further includes extracting data corresponding to the point where the indicator is inserted into the output to confirm that the indicator remains inserted before the output is displayed via the user interface. In such an embodiment, the user interface pipeline switching logic may include an extraction layer (e.g., the safe extractor 416) configured to extract the data. In some embodiments, program instructions (e.g., a trusted process 112A) receive the extracted data corresponding to the point where the indicator is inserted into the output and analyze the received extracted data to determine whether the indicator remains inserted into the output. In some embodiments, sensor pipeline switching logic processes data (e.g.,The sensor pipeline switching logic receives the sensor data (502) from the activated component and, in response to the indicator remaining inserted in the output, provides the processed data to a destination. In some embodiments, the sensor pipeline switching logic includes a dead man's switch (e.g., the disconnect switch 530 or 630) that periodically receives confirmation that the indicator remains inserted in the output and, in response to a lack of confirmation, interrupts the provision of the processed data to the destination. In some embodiments, the sensor pipeline switching logic implements an image sensor pipeline (e.g., the sensor pipelines 510) configured to process images received from a camera. In some embodiments, the sensor pipeline switching logic implements an audio sensor pipeline (e.g., the sensor pipelines 610) configured to process an audio signal received from a microphone.

[0079] Now with reference to Fig. Figure 11B shows a flowchart of method 1130. Method 1130 is an embodiment of a method performed by a data processing device, such as device 10, which includes a user interface and user interface pipeline switching logic. In many cases, performing method 1130 can significantly improve the security of the data processing device. In some embodiments, method 1130 may be implemented differently than shown.

[0080] Procedure 1130 begins in step 1135 with a user interface pipeline switching logic (e.g., a pipeline in the display unit 140, the audio unit 160, etc.) of the data processing device processing a set of data (e.g., the input data 402) to generate an output (e.g., the output single frame 404) for the user interface (e.g., a display, a speaker, etc.) of the data processing device. In step 1140, before displaying the output via the user interface, the user interface pipeline switching logic (e.g., via the safe extractor 416) extracts data from the output (e.g., the extracted pixel data 414) that corresponds to the point at which an indicator is inserted into the output to indicate that a component (e.g., a camera, a microphone, etc.) of the data processing device has been activated.In step 1145, the user interface pipeline switching logic provides the extracted data for analysis to determine whether the indicator remains included in the output. Example computer system

[0081] Now with reference to Fig. Figure 12 shows a block diagram illustrating an exemplary embodiment of a device 1200. In some embodiments, the device 1200 can implement functionality of the device 10. In some embodiments, elements of the device 1200 can be enclosed within a system-on-a-chip. In some embodiments, the device 1200 can be enclosed in a mobile data processing device, which may be battery-powered. Therefore, the power consumption of the device 1200 can be an important design consideration. In the illustrated embodiment, the device 1200 includes the fabric 102, a compute complex 1220 (which in some embodiments corresponds to the processor 110), a cache / memory control unit 1230, the display unit 140, the ISP 150, the audio unit 160, the neural engine 170, the graphics unit 180, and an input / output bridge (I / O bridge) 1260.In some embodiments, the device 1200 may include other components (not shown) in addition to or instead of the illustrated components, such as video processor encoders and decoders, image processing or recognition elements, computer vision elements, etc.

[0082] The Fabric 102 can include various intermediate connections, buses, MUXs, control units, etc., and can be configured to support communication between different elements of the Device 1200. In some embodiments, sections of the Fabric 102 can be configured to implement different communication protocols. In other embodiments, the Fabric 102 can implement a single communication protocol, and elements coupled to the Fabric 102 can internally convert from this single communication protocol to other communication protocols.

[0083] In the illustrated embodiment, the compute complex 1220 includes a bus interface unit (BIU) 1222, a cache 1224, and cores 1226A to B. In various embodiments, the compute complex 1220 can include different numbers of processors, processor cores, and caches. For example, the compute complex 1220 can include 1, 2, or 4 processor cores, or any other suitable number. In one embodiment, the cache 1224 is a set-associative L2 cache. In some embodiments, the cores 1226A to B can include internal instruction and / or data caches. In some embodiments, a coherence unit (not shown) can be configured in the fabric 102, the cache 1224, or elsewhere in the device 1200 to maintain coherence between different caches of the device 1200.The BIU 1222 can be configured to manage communication between the compute unit 1220 and other elements of the device 1200. Processor cores, such as cores 1226A to B, can be configured to execute instructions of a specific instruction set architecture (ISA), which may include operating system instructions and user application instructions. These instructions may be stored in a computer-readable medium, such as memory, coupled to the memory control unit 1230 discussed below.

[0084] As used herein, the term "coupled to" can indicate one or more connections between elements, and a coupling can include intervening elements. For example, in Fig. 12 the graphics unit 180 is described as being “coupled to” a memory by the Fabric 102 and the cache / memory control unit 1230. In contrast, in the illustrated embodiment of Fig. 12 the graphics unit 180 is “directly coupled” to the Fabric 102, as there are no intermediate elements.

[0085] The Cache / Memory Control Unit 1230 can be configured to manage the transfer of data between the Fabric 102 and one or more caches and storage devices. For example, the Cache / Memory Control Unit 1230 can be coupled to an L3 cache, which in turn can be coupled to system memory. In other embodiments, the Cache / Memory Control Unit 1230 can be directly coupled to storage. In some embodiments, the Cache / Memory Control Unit 1230 can include one or more internal caches. Any type of volatile memory coupled with the control unit 1230 can be used, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR, DDR2, DDR3, etc.) (including mobile versions of SDRAMs, such as mDDR3, etc., and / or low power consumption versions of SDRAMs, such as LPDDR4, etc.).), a RAMBUS DRAM (RDRAM), a static RAM (SRAM), etc. One or more memory devices can be coupled on a printed circuit board to form memory modules, such as Single Inline Memory Modules (SIMMs), Dual Inline Memory Modules (DIMMs), etc. Alternatively, the devices can be mounted with an integrated circuit in a chip-on-chip configuration, a package-on-package configuration, or a multi-chip module configuration. Memory coupled to the 1230 control unit can be any type of non-volatile memory, such as NAND flash memory, NOR flash memory, nano-RAM (NRAM), magnetoresistive RAM (MRAM), phase-change RAM (PRAM), racetrack memory, memristor memory, etc. As noted above, this memory can store program instructions, such as software blocks of . Fig. Store 2 and 3A to 3C, which can be executed by the computation complex 1220 to cause the data processing device 1200 to perform the functionality described herein.

[0086] The Graphics Unit 180 can include one or more processors, such as one or more graphics processing units (GPUs). The Graphics Unit 180 can receive graphics-oriented instructions, such as OPENGL®, Metal®, or DIRECT3D® instructions. The Graphics Unit 180 can execute specialized GPU instructions or perform other operations based on the received graphics-oriented instructions. The Graphics Unit 180 can be generally configured to process large blocks of data in parallel and can create images in a single-frame buffer for output to a display, which may be included in the device or be a separate device. The Graphics Unit 180 can include transformation, lighting, triangle, and / or rendering engines in one or more graphics processing pipelines. The Graphics Unit 180 can output pixel information for display images.The Graphics Unit 180, in various embodiments, can include programmable shader switching logic, which can include highly parallel execution cores configured to run graphics programs that may include pixel tasks, vertex tasks, and computational tasks (which may or may not be graphics-related).

[0087] The Display Unit 140 can be configured to read data from a single-frame buffer and provide a stream of pixel values ​​for display. In some embodiments, the Display Unit 140 can be configured as a display pipeline. Additionally, the Display Unit 140 can be configured to blend multiple single frames to produce a single output frame. Furthermore, the Display Unit 140 can include one or more interfaces (e.g., MIPI® or embedded DisplayPort (eDP)) for coupling to a user display (e.g., a touchscreen or external display).

[0088] The I / O bridge 1260 can include various elements configured to implement, for example, Universal Serial Bus (USB) communication, security, audio, and continuous operation with low power consumption. The I / O bridge 1260 can also include interfaces such as pulse width modulation (PWM), general-purpose input / output (GPIO), serial peripheral interface (SPI), and inter-integrated circuit (I2C). Various types of peripheral devices and equipment can be connected to the device 1200 via the I / O bridge 1260.

[0089] In some embodiments, the device 1200 includes (not explicitly shown) network interface switching logic that may be connected to the fabric 102 or the I / O bridge 1260. The network interface switching logic may be configured to communicate over various networks, which may be wired, wireless, or both. For example, the network interface switching logic may be configured to communicate over a wired local area network, a wireless local area network (e.g., via Wi-Fi™), or a wide area network (e.g., the Internet or a virtual private network). In some embodiments, the network interface switching logic is configured to communicate over one or more cellular networks that use one or more wireless access technologies.In some embodiments, the network interface switching logic is configured to communicate using device-to-device communication (e.g., Bluetooth® or Wi-Fi™ Direct), etc. In various embodiments, the Device 1200's network interface switching logic can provide connectivity to various types of other devices and networks. Application examples

[0090] Now with reference to Fig. 13. Various types of systems may include any of the circuits, devices, or systems discussed above. A system or device 1300 that incorporates or otherwise utilizes one or more of the techniques described herein may be used in a wide range of applications. For example, the system or device 1300 may be used as part of the hardware of systems such as a desktop computer 1310, a laptop computer 1320, a tablet computer 1330, a mobile phone 1340, or a television 1350 (or a set-top box connected to a television).

[0091] Similarly, disclosed elements can be used in a wearable device 1360, such as a smartwatch or a health monitoring device. Smartwatches, in many embodiments, can perform a variety of different functions, such as accessing emails, mobile services, calendars, health monitoring, etc. A wearable device can also be designed to perform exclusively health monitoring functions, such as monitoring a user's vital signs, performing epidemiological functions such as contact tracing, providing communication with an emergency medical service, etc.Other types of devices are also conceivable, including devices worn around the neck, devices that can be implanted in the human body, glasses or helmets that offer computer-generated reality experiences, such as those based on augmented and / or virtual reality, etc.

[0092] The system or device 1300 can also be used in various other contexts. For example, the system or device 1300 can be used in the context of a server computer system, such as a dedicated server, or on shared hardware implementing a cloud-based service 1370. Furthermore, the system or device 1300 can be implemented in a wide range of specialized everyday devices, including common household devices 1380, such as refrigerators, thermostats, security cameras, etc. The interconnection of such devices is often referred to as the "Internet of Things" (IoT). Elements can also be implemented in various modes of transportation. For example, the system or device 1300 could be used in the control systems, guidance systems, entertainment systems, etc., of various types of vehicles 1390.

[0093] The in Fig. The 13 illustrated applications are merely examples and are not intended to limit the possible future applications of disclosed systems or devices. Other application examples include, without limitation: portable gaming devices, music players, data storage devices, unmanned aerial vehicles, etc. Example of a computer-readable medium

[0094] In the present disclosure, various exemplary circuits have been described in detail above. It is intended that the present disclosure not only covers embodiments that include such switching logic, but also a computer-readable storage medium that includes design information specifying such switching logic. Accordingly, the present disclosure is intended to support claims that cover not only a device that includes the disclosed switching logic, but also a storage medium that specifies the switching logic in a format that is recognized by a data processing system configured to generate a simulation model of the hardware circuit, recognized by a manufacturing system configured to produce hardware (e.g., an integrated circuit) that includes the disclosed switching logic, and so on.Claims for such a storage medium should, for example, cover an entity that produces a circuit design but does not itself perform complete operations, such as: design simulation, design synthesis, circuit manufacturing, etc.

[0095] Now with reference to Fig. Figure 14 shows a block diagram of an exemplary non-volatile, computer-readable storage medium that stores circuit design information. In the illustrated embodiment, the data processing system 1440 is configured to process the design information. This may include executing instructions included in the design information, interpreting instructions included in the design information, compiling, transforming, or otherwise updating the design information, etc. Therefore, in some embodiments, the design information controls the data processing system 1440 (e.g., by programming the data processing system 1440) to perform various operations discussed below.

[0096] In the illustrated example, the data processing system 1440 processes the design information to generate both a computer simulation model 1460 of a hardware circuit and lower-level design information 1450. In other embodiments, the data processing system 1440 may generate only one of these outputs, may generate other outputs based on the design information, or both. With respect to the computational simulation, the data processing system 1440 may execute instructions of a hardware description language that includes register-transfer-level (RTL) code, behavioral code, structure code, or a combination thereof. The simulation model may perform the functionality specified by the design information, enable verification of the functional correctness of the hardware design, generate power consumption estimates, generate timing estimates, and so on.

[0097] In the illustrated example, the data processing system 1440 also processes the design information to generate lower-level design information 1450 (e.g., gate-level design information, a netlist, etc.). This can include synthesis operations, as shown, such as constructing a multi-level network, optimizing the network using technology-independent or technology-dependent techniques, or both, and outputting a network of gates (with possible constraints based on gates available in a technology library, dimensioning, delay, current, etc.). Based on the lower-level design information 1450 (possibly alongside other inputs), the semiconductor manufacturing system 1420 is configured to fabricate an integrated circuit 1430 (which may correspond to the functionality of the simulation model 1460).It should be noted that the data processing system 1440 can generate various simulation models based on design information at different levels of description, including information 1450, 1415, and so on. The data representing the design information 1450 and the model 1460 can be stored on medium 1410 or on another medium or several other media.

[0098] In some embodiments, the design information of a lower level 1450 controls (e.g., programs) the semiconductor manufacturing system 1420 to manufacture the integrated circuit 1430. When the design information is processed by the manufacturing system, it can thus program the manufacturing system to produce a circuit incorporating various switching logics disclosed herein.

[0099] The non-volatile computer-readable storage medium 1410 can comprise any of several suitable types of storage devices or storage devices. The non-volatile computer-readable storage medium 1410 can be an installation medium, such as a CD-ROM, floppy disks, or a tape device; computer system memory or random-access memory, such as DRAM, DDR-RAM, SRAM, EDO-RAM, Rambus-RAM, etc.; non-volatile memory, such as flash memory; magnetic media, such as a hard disk drive or optical storage; registers or other similar types of memory elements, etc. The non-volatile computer-readable storage medium 1410 can also include other types of non-volatile memory or combinations thereof.Accordingly, the non-volatile, computer-readable storage medium 1410 can include two or more storage media; such media can be located in different places - for example, in different computer systems connected via a network.

[0100] The design information 1415 can be specified using any of several suitable computer languages, including, but not limited to, hardware description languages ​​such as VHDL, Verilog, SystemC, SystemVerilog, RHDL, M, MyHDL, etc. The format of various design information can be recognized by one or more applications executed by the data processing system 1440, the semiconductor manufacturing system 1420, or both. In some embodiments, the design information may also include one or more cell libraries that specify the synthesis, layout, or both of the integrated circuit 1430. In some embodiments, the design information is specified wholly or partly in the form of a netlist that specifies the cell library elements and their connectivity.The design information discussed herein may, on its own, contain sufficient information for the fabrication of a corresponding integrated circuit, but it need not. For example, the design information may specify the circuit elements to be manufactured, but not their physical layout. In this case, design information can be combined with layout information to actually manufacture the specified circuit logic.

[0101] The 1430 integrated circuit, in various embodiments, can include one or more user-defined macrocells, such as memory, analog or mixed-signal circuits, and the like. In such cases, the design information can include information relating to the included macrocells. Such information can include, but is not limited to, a schematic capture database, mask design data, behavioral models, and device- or transistor-level netlists. Mask design data can be formatted according to the Graphics Data System (GDSII) or any other suitable format.

[0102] The Semiconductor Manufacturing System 1420 can include any of several suitable elements configured to manufacture integrated circuits. These can include, for example, elements for depositing semiconductor materials (e.g., onto a wafer, which may include masking), removing materials, changing the shape of deposited materials, modifying materials (e.g., by doping materials or modifying dielectric constants using ultraviolet processing), and so on. The Semiconductor Manufacturing System 1420 can also be configured to perform various tests on manufactured circuits to ensure proper function.

[0103] In various embodiments, the integrated circuit 1430 and model 1460 are configured to operate according to a circuit design specified by design information 1415, which may include performing any of the functionality described herein. For example, the integrated circuit 1430 may perform any of the various functions described in Fig. Include the elements shown in points 1 to 8. Furthermore, the 1430 integrated circuit can be configured to perform various functions described herein in conjunction with other components. Additionally, the functionality described herein can be performed by multiple interconnected integrated circuits.

[0104] As used herein, an expression of the form "design information specifying a design of a circuit configured to..." does not imply that the circuit in question must be manufactured for the element to be satisfied. Rather, this expression indicates that the design information describes a circuit that, after manufacturing, will be configured to perform the specified actions or to include the specified components. Similarly, the statement "instructions of a hardware description programming language that are 'executable' to program a data processing system to generate a computer simulation model" does not imply that the instructions must be executed for the element to be satisfied, but rather indicates properties of the instructions.Additional features relating to the model (or the circuit represented by the model) can similarly relate to properties of the instructions in this context. Therefore, an entity that sells a computer-readable medium containing instructions that meet the specified properties may be supplying an infringing product, even if another entity actually executes the instructions on the medium.

[0105] It is important to note that a given design, at least in the context of digital logic, can be implemented using a wide variety of different gate arrangements, circuit technologies, and so on. However, once a digital logic design is specified, professionals do not need to conduct significant experimentation or research to determine these implementations. Rather, they are familiar with procedures that reliably and predictably produce one or more circuit implementations that provide the functionality described in the design information. The different circuit implementations may affect the performance, area, power consumption, and other aspects of a given design (possibly involving trade-offs between different design goals), but the logical functionality does not vary between the different circuit implementations of the same circuit design.

[0106] In some embodiments, the instructions included in the design information provide RTL information (or other higher-level design information) and are executable by the data processing system to synthesize a gate-level netlist that represents the hardware circuit based on the RTL information as input. Similarly, the instructions can provide behavioral information and be executable by the data processing system to synthesize a netlist or other lower-level design information. The lower-level design information can program the manufacturing system 1420 to manufacture the integrated circuit 1430.

[0107] The present disclosure includes references to “an embodiment” or groups of “embodiments” (e.g., “some embodiments” or “various embodiments”). Embodiments are different implementations or instances of the disclosed concepts. References to “embodiment,” “an embodiment,” “a particular embodiment,” and the like do not necessarily refer to the same embodiment. A large number of possible embodiments are considered, including those specifically disclosed, as well as modifications or alternatives that fall within the nature or scope of protection of the disclosure.

[0108] This disclosure may discuss potential advantages that may arise from the disclosed embodiments. Not all implementations of these embodiments necessarily exhibit any or all potential advantages. Whether an advantage is achieved for a particular implementation depends on many factors, some of which are outside the scope of this disclosure. Indeed, there are several reasons why an implementation that falls within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation might include different switching logic outside the scope of the disclosure, which, in conjunction with one of the disclosed embodiments, eliminates or reduces one or more of the disclosed advantages. Furthermore, a suboptimal design implementation of a particular implementation (e.g.,Implementation techniques or tools) negate or diminish disclosed advantages. Even assuming a qualified implementation, the attainment of advantages may still depend on other factors, such as the environmental circumstances in which the implementation is deployed. For example, inputs provided to a particular implementation may prevent one or more problems addressed in this disclosure from occurring on a particular occasion, thereby potentially preventing the advantage of its solution from being achieved. Due to the existence of possible factors outside this disclosure, it is expressly intended that all potential advantages described herein are not to be construed as limitations on claims that must be satisfied to prove infringement.Rather, the identification of such potential benefits is intended to illustrate the type(s) of improvement available to designers who benefit from this disclosure. The fact that such benefits are described in a permissive manner (e.g., by stating that a certain benefit “may occur”) is not intended to cast doubt on whether such benefits can actually be achieved, but instead to acknowledge the technical reality that achieving such benefits often depends on additional factors.

[0109] Unless otherwise stated, embodiments are not limiting. This means that the disclosed embodiments are not intended to limit the scope of protection of claims formulated on the basis of this disclosure, even if only a single example is described with respect to a particular feature. The disclosed embodiments are intended to be illustrative and not limiting, unless the disclosure contains statements to the contrary. The application is thus intended to allow the application to cover embodiments disclosed in claims as well as the alternatives, modifications, and equivalents that are apparent to a person skilled in the art who benefits from this disclosure.

[0110] For example, features in this application may be combined in any suitable way. Accordingly, during the further pursuit of this application (or an application claiming priority thereof), new claims may be formulated to any such combination of features. In particular, with reference to the accompanying claims, features from dependent claims may be combined with those of other dependent claims where appropriate, including claims dependent on other independent claims. Similarly, features from respective independent claims may be combined as appropriate.

[0111] While the accompanying dependent claims may be formulated such that each depends on a single other claim, additional dependencies are also considered. All combinations of features in the dependents that are consistent with this disclosure are considered and may be claimed in this or any other application. In summary, combinations are not limited to those specifically enumerated in the accompanying claims.

[0112] Where appropriate, consideration will also be given to ensuring that claims formulated in one format or statutory type (e.g., establishment) support corresponding claims in another format or statutory type (e.g., procedure).

[0113] Because this disclosure is a legal document, various terms and expressions may be subject to regulatory and legal interpretation. It is hereby announced that the following paragraphs, as well as definitions provided throughout this disclosure, should be used in determining how claims formulated based on this disclosure are to be interpreted.

[0114] References to a singular form of an element (i.e., a noun or noun phrase preceded by "a" or "the") should, unless the context clearly indicates otherwise, mean "one or more." Thus, a reference to "an element" in a claim, without accompanying context, does not exclude additional instances of the element. A "multitude" of elements refers to a set of two or more of the elements.

[0115] The word “can / can” is used here in a permissive sense (i.e. having the potential to be able to) and not in an obligatory sense (i.e. must / must).

[0116] The terms “comprehensive” and “inclusive” and forms thereof are open and mean “including without being limited to”.

[0117] When the term “or” is used in this disclosure in relation to a list of options, it is generally understood to be used in an inclusive sense, unless the context indicates otherwise. Thus, a statement of “x or y” is equivalent to “x or y or both” and therefore covers 1) x but not y, 2) y but not x, and 3) both x and y. On the other hand, an expression such as “either x or y, but not both” makes it clear that “or” is used in an exclusive sense.

[0118] A statement of "w, x, y, or z, or any combination thereof" or "at least one of ... w, x, y, and z" is intended to cover all possibilities involving a single element up to the total number of elements in the sentence. For the sentence [w, x, y, z], for example, these expressions cover each individual element of the sentence (e.g., w, but not x, y, or z), any two elements (e.g., w and x, but not y or z), any three elements (e.g., w, x, and y, but not z), and all four elements. The expression "at least one of ... x, y, and z" thus refers to at least one element of the sentence [w, x, y, z], thereby covering all possible combinations in this list of elements. This expression must not be interpreted as requiring the presence of at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0119] Various “labels” may appear before nouns or noun phrases in this disclosure. Unless the context indicates otherwise, different labels used for a feature (e.g., “first circuit,” “second circuit,” “particular circuit,” “given circuit,” etc.) refer to different instances of the feature. Additionally, when applied to a feature, the labels “first,” “second,” and “third” do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise specified.

[0120] The phrase "based on" is used to describe one or more factors that influence an investigation. This term does not preclude the possibility that additional factors may influence the investigation. That is, an investigation may be based solely on specified factors, or on the specified factors as well as other, unspecified factors. Consider the phrase "to determine A based on B." This phrase indicates that B is a factor used to determine A, or that influences the determination of A. This phrase does not preclude the possibility that the determination of A may also be based on another factor, such as C. This phrase is also intended to cover an embodiment in which A is determined solely based on B.As used herein, the expression “based on” is synonymous with the expression “based at least in part on”.

[0121] The expressions "in response to" and "in reaction to" describe one or more factors that trigger an effect. This expression does not preclude the possibility that additional factors may influence or otherwise trigger the effect, either together with the stated factors or independently of them. That is, an effect may occur solely in response to these factors, or it may occur in response to the stated factors as well as other, unspecified factors. Consider the expression "to perform A in response to B." This expression indicates that B is a factor that triggers the performance of A or that triggers a particular outcome for A. This expression does not preclude the possibility that performing A may also occur in response to another factor, such as C. Nor does it preclude the possibility that performing A may occur in response to both B and C.This expression is also intended to cover an embodiment in which A is carried out solely in response to B. As used herein, the expression "responding to" is synonymous with the expression "at least partially responding to". Similarly, the expression "in response to" is synonymous with the phrase "at least partially in response to".

[0122] Within this disclosure, different entities (which may be variously referred to as "units," "circuits," other components, etc.) may be described or claimed to be "configured" to perform one or more tasks or operations. This phrase—[entity] configured to [perform one or more tasks]—is used herein to refer to a structure (i.e., something physical). In particular, this phrase is used to indicate that this structure is arranged to perform the one or more tasks during operation. A structure may be described as "configured to" perform a task even if the structure is not currently operating.Thus, an entity described or specified as "configured to" perform a task refers to something physical, such as a device, a circuit, a system with a processing unit and memory that stores program instructions executable to implement the task, etc. This expression is not used herein to refer to something immaterial.

[0123] In some cases, various units / circuits / components herein may be described as performing a set of tasks or operations. It is understood that these entities are "configured to" perform these tasks / operations, even if this is not specifically stated.

[0124] The term "configured to" should not be interpreted as "configurable to." For example, an unprogrammed FPGA would not be considered "configured to" perform a specific function. However, this unprogrammed FPGA may be "configurable to" perform that function. After appropriate programming, the FPGA can then be described as "configured to" perform the specific function.

[0125] For the purposes of U.S. patent applications based on this disclosure, a claim that a structure is "configured to" perform one or more functions shall expressly not rely on the application of 35 U.S.C. § 112(f) for that claim element. If, during the grant proceedings of a U.S. patent application based on this disclosure, the applicant wishes to rely on the application of section 112(f), they shall state claim elements using the construct "means to" [perform a function].

[0126] Various “circuits” can be described in this disclosure. These circuits, or “switching logics,” constitute hardware that includes various types of circuit elements, such as combinational logic, clocked storage devices (e.g., flip-flops, registers, latching circuits, etc.), finite state machines, memory (e.g., random-access memory, embedded dynamic random-access memory), programmable logic arrays, and so on. The switching logic may be user-defined or derived from standard libraries. In different implementations, switching logic may include digital components, analog components, or a combination of both. Certain types of circuits may be referred to more generally as “units” (e.g., a decoding unit, an arithmetic logic unit (ALU), a functional unit, a memory management unit (MMU), etc.).Such units also refer to circuits or switching logics.

[0127] The disclosed circuits / units / components and other elements illustrated in the drawings and described herein thus include hardware elements such as those described in the preceding paragraph. In many cases, the internal arrangement of hardware elements within a particular circuit can be specified by describing the function of that circuit. For example, a particular "decoding unit" can be described as performing the function of "processing an opcode of an instruction and redirecting that instruction to one or more of a plurality of functional units," meaning that the decoding unit is "configured to" perform this function. This functional description is sufficient for a person skilled in the art in computing to further define a set of possible structures for the circuit.

[0128] In various embodiments, as discussed in the preceding paragraph, circuits, units, and other elements can be defined by the functions or operations for which they are configured. The arrangement of these circuits / units / components in relation to one another and the way in which they interact constitute a microarchitectural definition of the hardware, which is ultimately fabricated in an integrated circuit or programmed into an FPGA to form a physical implementation of the microarchitectural definition. Thus, the microarchitectural definition is recognized by the person skilled in the art as a structure from which many physical implementations can be derived, all of which fall within the broader structure described by the microarchitectural definition.This means that the person skilled in the art, to whom the microarchitecture definition provided according to this disclosure is presented, can implement the structure without undue experimentation and by applying average skills by encoding the description of the circuits / units / components in a hardware description language (HDL) such as Verilog or VHDL. The HDL description is often expressed in a way that may appear functional. However, to the person skilled in the art, this HDL description is the way in which the structure of a circuit, unit, or component is transformed to the next level of implementation detail. Such an HDL description may take the form of behavioral code (which is not usually synthesizable), register-transfer language (RTL) code (which, unlike behavioral code, is usually synthesizable), or structural code (e.g.,a netlist specifying logic gates and their connectivity). The HDL description can then be synthesized against a library of cells designed for a given integrated circuit fabrication technology and modified for timing, power, and other reasons to result in a final design database that is sent to a foundry to generate masks and ultimately fabricate the integrated circuit. Some hardware circuits or sections thereof can also be custom-designed in a schematic editor and incorporated into the integrated circuit design along with synthesized switching logic. The integrated circuits can include transistors and other circuit elements (e.g., passive elements such as capacitors, resistors, inductors, etc.).) and establish a connection between the transistors and circuit elements. Some embodiments can implement multiple integrated circuits coupled together to implement the hardware circuitry, and / or discrete elements can be used in some embodiments. Alternatively, the HDL design can be synthesized into a programmable logic array such as a field-programmable gate array (FPGA) and implemented within the FPGA.This decoupling between the design of a group of circuits and the subsequent low-level implementation of those circuits typically leads to the scenario in which the circuit or logic designer never specifies a particular set of structures for low-level implementation beyond a description of what the circuit is configured for, as this process is carried out at a different stage of the circuit implementation process.

[0129] The fact that many different low-level combinations of circuit elements can be used to implement the same circuit specification leads to a large number of equivalent structures for that circuit. As indicated, these low-level circuit implementations can vary according to changes in the manufacturing technology, the fabrication equipment chosen to manufacture the integrated circuit, the library of cells provided for a particular project, and so on. In many cases, the choices made by different design tools or methodologies for fabricating these different implementations can be arbitrary.

[0130] Furthermore, for a single implementation of a particular functional specification of a circuit, it is common to include a large number of devices (e.g., millions of transistors) for a given embodiment. Accordingly, the sheer volume of this information makes it impractical to provide a complete low-level specification of the structure used to implement a single embodiment, let alone the vast array of equivalent possible implementations. For this reason, the present disclosure describes a structure of circuits using the functional shorthand notation commonly employed in industry.

[0131] Various embodiments of a safe environment, enhanced by the implementation of safe isolated areas, are considered based on the foregoing description, including, but not limited to, the embodiments listed below. 1. A data processing device comprising: a sensor and a sensor processor switching logic that is coupled with and configured to the sensor for: Processing sensor data received from a sensor of the data processing device; In response to an initial indication that a first consumer is trusted, provide an initial dataset of the processed sensor data for the first consumer and In response to a second indication that a second consumer is not trustworthy, negotiating one or more conditions under which the second consumer is allowed to receive a second set of processed sensor data. 2. The data processing device according to embodiment 1, wherein the first specification identifies the first consumer as being located in a secure environment in which a set of security criteria is enforced for the first data record; and wherein the second specification identifies the second consumer as being located outside the secure environment. 3. The data processing device according to embodiment 2, further comprising: one or more processors; and a memory containing program instructions that can be executed by one or more processor(s) for: Implementing a safe execution environment, wherein the first consumer is a first process that is executed within the safe execution environment, and wherein the second consumer is a second process that is executed outside the safe execution environment. 4. The data processing device according to embodiment 2, comprising the sensor processor switching logic: one or more configuration registers that are configured to: Storing configuration information that controls the operation of the sensor, where one or more configuration registers are only addressable by entities within the secure environment. 5. The data processing device according to embodiment 1, wherein the sensor processor switching logic is an image signal processor configured to process sensor data received from a camera. 6. The data processing device according to embodiment 1, wherein the sensor processor switching logic is an audio processor configured to process sensor data received from a microphone. 7. The data processing device according to embodiment 1, wherein the sensor processor switching logic is configured to: Providing an indication that the sensor is active, wherein one or more conditions include that the sensor processor switching logic receives confirmation that a user will be notified that the sensor is active. 8. The data processing device according to embodiment 7, further comprising: a display pipeline switching logic configured to: In response to the provided information, pixel data is inserted into a single image that is displayed on a screen to notify the user that the sensor is active. 9. The data processing device according to embodiment 1, wherein the sensor processor switching logic is configured to: in response to the determination to provide the second data set to the second consumer: periodic reception of a heartbeat signal indicating that one or more conditions have been met; and Terminating the provision of the second data set in response to the determination that the heartbeat signal is no longer being received. 10. The data processing device according to embodiment 1, comprising the sensor signal processor switching logic: a switch that is configured to: In response to one or more conditions being violated, interrupting a data path through which the second data set is provided to the second consumer. 11. The data processing device according to embodiment 1, wherein the sensor processor switching logic is configured to: Switching off the sensor in response to the determination that one or more conditions have been violated. 12. The data processing device according to embodiment 1, comprising the sensor signal processor switching logic: a secure pipeline configured to: Processing sensor data to generate the first data set for the first consumer; and an insecure pipeline configured to: Processing sensor data to generate the second data set for the second consumer. 13. The data processing device according to embodiment 12, comprising the sensor signal processor switching logic: an input / output memory management unit (IOMMU) configured to: Storing a first set of memory addresses designated as accessible to the secure pipeline and a second set of memory addresses designated as accessible to the unsafe pipeline; and Preventing the safe pipeline from accessing memory addresses outside the first set, and preventing the unsafe pipeline from accessing memory addresses outside the second set. 14. The data processing device according to embodiment 13, wherein the memory addresses are stored as translations from virtual to physical addresses. 15. The data processing device according to embodiment 1, comprising the sensor signal processor switching logic: a first direct memory access (DMA) engine configured to write the first record to a memory section accessible to the first consumer; and a second DMA engine configured to write the second data to a different memory section accessible to the second consumer. 16. A comprehensive procedure: Processing, by means of a sensor-processor switching logic of a data processing device, of sensor data received from a sensor of the data processing device; In response to an initial indication that a first consumer is in a safe environment, the sensor processor switching logic provides an initial data set of the processed sensor data to the first consumer and In response to a second indication that a second consumer is located outside the safe environment, the sensor processor switching logic negotiates one or more conditions under which the second consumer is allowed to receive a second set of processed sensor data. 17. The method according to embodiment 16, wherein the negotiated one condition or the negotiated multiple conditions include a condition that a user is notified of the use of the sensor. 18. An integrated circuit comprising: one or more processors configured to: Joint execution of trusted processes and untrusted processes in an isolated manner, which hinders enforcement includes a set of safety criteria; and a sensor processor switching logic that is configured to: Processing sensor data received from a sensor; In response to an initial indication that a first process is one of the trusted processes, providing an initial dataset of the processed sensor data to the first process and In response to a second indication that a second process is one of the untrusted processes, restrict, based on one or more of the set of security criteria, access by the second process to a second dataset of the processed sensor data. 19. The integrated circuit according to embodiment 18, comprising the sensor processor switching logic: a secure pipeline configured to: Processing sensor data to generate the first data set for the first process; and an input / output memory management unit (IOMMU) configured to: Storing a set of memory addresses designated as accessible to the secure pipeline; and Prevent the secure pipeline from accessing memory addresses outside the set. 20. The integrated circuit according to embodiment 18, further comprising: a user interface pipeline switching logic that is configured to: Inserting an indicator that the sensor is active into an output provided to a user interface; and where the sensor processor switching logic is configured to: Based on the indicator meeting one or more of the set of security criteria, granting access to the second dataset of processed sensor data for the second process. 21. A data processing device comprising: a user interface and a user interface pipeline switching logic that is coupled with and configured to the user interface to: Processing a set of data received from an initial source to produce an output for the user interface of the data processing device; Received from a second source, an indication that a component of the data processing device has been activated; and Before displaying the output via the user interface, insert an indicator of the activated component into the output. 22. The data processing device according to embodiment 21, wherein the component is a sensor configured to collect sensitive data about a user. 23. The data processing device according to embodiment 21, wherein the component is a camera configured to collect image data. 24. The data processing device according to embodiment 21, wherein the component is a microphone configured to capture audio data. 25. The data processing device according to embodiment 21, wherein the user interface is a display and wherein the user interface pipeline switching logic is a display pipeline switching logic configured to: Generating individual images for display on the data processing device, wherein the display pipeline switching logic includes a mixing pipeline stage configured to: Based on the received information, pixel data is inserted as the indicator into a single image that is rendered by the display pipeline switching logic. 26. The data processing device according to embodiment 21, wherein the user interface is a loudspeaker and wherein the user interface pipeline switching logic is an audio pipeline switching logic configured to: Generating an audio signal for the loudspeaker of the Data processing device, wherein the audio pipeline switching logic includes a mixing pipeline stage configured to: Based on the received information, insert an audio indicator into the audio signal. 27. The data processing device according to embodiment 21, wherein the user interface is a haptic feedback engine and wherein the user interface pipeline switching logic is a haptic pipeline switching logic configured to: Generating haptic feedback data for a haptic feedback engine, where the haptic pipeline switching logic includes a mixing pipeline stage configured to: Based on the received information, a specific haptic indicator is inserted into the haptic feedback data. 28. The data processing device according to embodiment 21, further comprising: one or more processors; and a memory containing program instructions that can be executed by one or more processors for: Providing, via an initial untrusted process corresponding to the first source, the set of data to the user interface pipeline switching logic; and Provide, via a second trusted process that corresponds to the second source, the specification to the user interface pipeline switching logic. 29. The data processing device according to embodiment 21, wherein the user interface pipeline includes switching logic: a first direct memory access (DMA) engine configured to retrieve the set of data from memory; and a second, different DMA engine that is configured to retrieve the information from memory. 30. The data processing device according to embodiment 21, wherein the user interface pipeline switching logic includes an extraction stage configured to: Extracting data corresponding to the point where the indicator is inserted into the output to confirm that the indicator remains inserted into the output before the output is displayed via the user interface. 31. The data processing device according to embodiment 30, further comprising: one or more processors; and a memory containing program instructions that can be executed by one or more processors for: Receiving the extracted data corresponding to the point where the indicator is inserted into the output; and Analyze the received extracted data to determine whether the indicator should remain in the output. 32. The data processing device according to embodiment 30, further comprising: a sensor pipeline switching logic configured to: Processing data received from the activated component; and In response to the indicator remaining included in the output, the processed data is provided to a destination. 33. The data processing device according to embodiment 32, comprising the sensor pipeline switching logic: a dead man's switch configured to: periodic receipt of confirmation that the indicator remains included in the output; and In response to a lack of confirmation, the provision of the processed data to the destination is interrupted. 34. The data processing device according to embodiment 32, wherein the sensor pipeline switching logic implements an image sensor pipeline configured to process images received from a camera. 35. The data processing device according to embodiment 32, wherein the sensor pipeline switching logic implements an audio sensor pipeline configured to process an audio signal received from a microphone. 36. A data processing device comprising: a user interface; and a user interface pipeline switching logic that is coupled with and configured to the user interface to: Processing a set of data to generate an output for the user interface of the data processing device; Extracting, from the output prior to its display via the user interface, data corresponding to the point where an indicator is inserted into the output to indicate that a component of the data processing device has been activated; and Providing the extracted data for analysis to determine whether the indicator should remain in the output. 37. The data processing device according to embodiment 36, further comprising: one or more processors; and a memory containing program instructions that can be executed by one or more processors for: Determine, from the provided extracted data, whether the indicator should remain in the output; and In response to determining that the indicator should not remain included in the output, causing an interruption of an output associated with the indicator. 38. The data processing device according to embodiment 36, further comprising: a sensor and a sensor pipeline switching logic configured to: Processing data received from the sensor; and in response to the indicator remaining included in the output, Providing the processed data to a destination. 39. The data processing device according to embodiment 36, wherein the user interface pipeline includes switching logic: a first direct memory access (DMA) engine configured to retrieve the set of data for processing; and a second, different DMA engine that is configured to provide the extracted data to a target for analysis. 40. A comprehensive procedure: Processing, through a user interface pipeline switching logic of a data processing device, a set of data received from a source to generate an output for a user interface of the data processing device; before displaying the output via the user interface: Inserting, through a first stage of the user interface pipeline switching logic into the output, an indicator of an activated component and Extract, through a second stage of the user interface pipeline switching logic, from the output data corresponding to the point where the indicator is inserted, to confirm that the indicator remains inserted.

[0132] Numerous variations and modifications become apparent to the person skilled in the art once the foregoing disclosure is fully understood. It is intended that the following claims be interpreted to include all such variations and modifications.

Claims

[1] Data processing device, comprising: one or more processors configured to: Joint execution of trusted processes and untrusted processes in an isolated manner, which includes implementing a secure environment in which a set of security criteria is enforced for data of the trusted processes; and multiple heterogeneous hardware accelerators configured to: Implementing isolated areas of the secure environment that extend the enforcement of one or more of the set of security criteria within the hardware accelerators for data distributed to the hardware accelerators for the performance of tasks assigned to the trusted processes. [2] Data processing device according to claim 1, wherein, in order to extend the enforcement of one or more security criteria, one or more of the hardware accelerators is / are further configured to: Preventing tasks assigned to untrusted processes from accessing memory regions assigned to trusted processes. [3] Data processing device according to claim 2, wherein the one or more hardware accelerators are further configured to prevent the tasks of: Storing translations from virtual to physical addresses for the memory regions allocated to trusted processes and for memory regions allocated to untrusted processes; and Prevent tasks assigned to untrusted processes from accessing the translations of virtual to physical addresses for the memory regions allocated to trusted processes. [4] Data processing device according to claim 2, wherein the one or more hardware accelerators include a graphics processing unit (GPU) configured to: Maintain a table that identifies the memory regions allocated to trusted processes and the memory regions allocated to untrusted processes; and Perform memory accesses according to the table. [5] Data processing device according to claim 1, wherein, to extend the enforcement of one or more security criteria, one or more of the hardware accelerators is / are further configured to: Physically isolating distributed data associated with trusted processes from distributed data associated with untrusted processes. [6] Data processing device according to claim 5, wherein the one or more hardware accelerators include pipelines configured to: Performing tasks assigned to the untrusted processes and tasks assigned to the trusted processes, wherein the pipelines include one or more additional pipeline stage(s) that process the distributed data assigned to the untrusted processes and do not process the distributed data assigned to the untrusted processes. [7] Data processing device according to claim 6, wherein the one or more hardware accelerators include a display unit having a display pipeline configured to: Rendering individual frames for a display of Data processing device, wherein the display pipeline includes a mixing pipeline stage configured to: Mixing pixel data received from a trusted process into a single image that is rendered through the display pipeline and includes pixel data from an untrusted process. [8] Data processing device according to claim 6, wherein the one or more hardware accelerators include a display unit having a display pipeline configured to: Rendering of individual images for a display of the data processing device, wherein the display pipeline includes an extraction pipeline stage configured to: Extracting pixel data from a single image being rendered through the display pipeline; and Providing the extracted pixel data to a trusted process. [9] Data processing device according to claim 5, wherein the one or more hardware accelerators include one or more additional data buffers configured to: Storing the distributed data associated with the trusted processes, with one or more additional buffers not storing the distributed data associated with the untrusted processes. [10] Data processing device according to claim 9, wherein the one or more hardware accelerators include / include a neural engine configured to: Performing a set of operations of a neural network, where the neural engine includes one or more data buffers to store distributed data for tasks assigned to trusted processes. [11] Data processing device according to claim 1, wherein, to extend the enforcement of one or more security criteria, one or more of the hardware accelerators is / are further configured to: Preventing sensor data provided to trusted processes from being provided to untrusted processes. [12] Data processing device according to claim 11, wherein the one or more hardware accelerators include / include an image signal processor configured to: Processing sensor data received from a camera sensor, wherein the image signal processor includes a disconnect switch configured to enable or disable the provision of the sensor data to an untrusted process in response to the fulfillment of one or more security criteria. [13] Data processing device according to claim 12, wherein the one or more security criteria includes / include an indication present in a single image that is displayed to a user to indicate that the camera sensor is currently in use. [14] Data processing device according to claim 11, wherein the one or more hardware accelerators include / include an audio unit configured to: Processing sensor data received from a microphone sensor, wherein the audio unit includes a disconnect switch configured to enable or disable the provision of sensor data to an untrusted process in response to one or more security criteria being met. [15] Data processing device according to claim 1, further comprising: a system-on-a-chip (SoC) that includes one or more processors and one or more hardware accelerators. [16] Data processing device comprising: one or more processors; a variety of heterogeneous hardware accelerators and a memory containing program instructions that can be executed by one or more processor(s) for: Isolating jointly executed trusted processes and untrusted processes and Distributing data to some of the many heterogeneous hardware accelerators to perform tasks requested by the processes; and where the heterogeneous hardware accelerators are configured to: Receiving information about a method that isolates trusted and untrusted processes; and Based on the information received, extending the isolation of trusted and untrusted processes for jointly executed tasks that work on the distributed data. [17] Data processing device according to claim 16, wherein, to further extend the isolation of the trusted and the untrusted processes, one or more of the hardware accelerators is / are further configured to: Storing translations from virtual to physical addresses for memory regions allocated to trusted processes and for memory regions allocated to untrusted processes; and Prevent tasks assigned to untrusted processes from accessing the translations of virtual to physical addresses for the memory regions allocated to trusted processes, in order to prevent tasks assigned to untrusted processes from accessing memory regions allocated to trusted processes. [18] Data processing device according to claim 16, wherein, to further extend the isolation of the trusted and the untrusted processes, one or more of the hardware accelerators is / are further configured to: Physically isolating distributed data associated with trusted processes from distributed data associated with untrusted processes. [19] Data processing device comprising: one or more processors configured to: Joint execution of trusted and untrusted processes in such a way that the trusted processes are isolated from the untrusted processes; and multiple heterogeneous hardware accelerators configured to: Performing tasks required by the trusted processes; and Negotiating conditions under which it is permitted to carry out tasks requested by the untrusted processes. [20] Data processing device according to claim 19, wherein one of the negotiated conditions includes a notification to a user indicating a task requested by one of the untrusted processes.

Images