Virtualization of image signal processor hardware

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for virtualizing image signal processing (ISP). A method can include assigning a first virtual machine (VM) to a first image capture device and a second VM to a second image capture device; allocating a first portion of the device memory to the first VM and a second portion of the device memory to the second VM; storing processing parameters of a first ISP block in the first portion of the device memory; providing data indicating a location of the first portion of the device memory to a microcontroller coupled to the plurality of ISP blocks; transmitting the processing parameters from the first portion of the device memory to ISP registers corresponding to the first ISP block; and processing image data from the first image capture device using the processing parameters.

Description

[0001] PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0002] Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT

[0003] VIRTUALIZATION OF IMAGE SIGNAL PROCESSOR HARDWARE

[0004] BACKGROUND

[0005] Image processing can be resource intensive. For image processing performed on smartphones, resources may be constrained, e.g., due to limited power resources or desire for increased battery life. Such devices may perform image processing using dedicated hardware blocks, such as image signal processing (ISP) hardware. Image processing can include adjustments to raw image data obtained using an image capture device, e.g., to improve the quality of images taken or enable specific tasks to be performed using the raw image data.

[0006] SUMMARY

[0007] This specification describes technologies for virtualizing image signal processing (ISP). In some cases, a set of virtual machines can be uniquely assigned to one or more cameras. A microcontroller can control data access to memory by ISP blocks operating for the set of virtual machines, e.g., to ensure that image data for one virtual machine is correctly processed for that virtual machine and not influenced or affected by other virtual machines configured for processing other image data. Example use cases include obtaining image streams from multiple cameras and having multiple processes running that require data from the image streams, such as a front-facing camera capturing a front facing view and a rear-facing camera capturing a rear facing view. In some cases, any of one or more cameras for face detection, one or more infrared (IR) cameras for face detection or facialbased security', one or more cameras for biometric authentication, such as fingerprint scanning, or one or more cameras for ambient environment detection, among other types of cameras, can be processed in sub-groups or individually using different virtual machines.

[0008] The technology described in this specification can be implemented so as to realize one or more of the following technical advantages. For example, these techniques can prevent processing of multiple camera streams from interfering with one another. Interference can include incidental or intentional interference. Incidental interference can include process bugs that cause, e.g., an image processing element to request image data from a second camera to be processed with processing parameters generated for a first camera based on a bug that incorrectly identified the image data to be requested. Intentional interference can include malicious processes, such as processes that attempt to inject data PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0009] Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT to spoof an image security detection process or ransom a device through denial of an imaging device.

[0010] Techniques can prevent interference by, e.g., using a multi-step data obtaining and processing pipeline as described further in this document. For example, a memory management unit can check and control memory requests for read or write actions. In some cases, a unique identifier can be generated for each camera or each camera data stream. Data, including raw image data, processed image data, processing parameter data, or a combination of these, can be stored with the identifier to enable a memory management unit to check and confirm or deny read and write requests, e.g., for any portion of the data.

[0011] Techniques can result in reduced power consumption compared to pure software approaches. For example, techniques, including memory management, can be performed in some cases by software running on a processing device. Such software can include programs running on a CPU to obtain raw image data or other data and determine, based on the obtained data, how to process one or more image data streams. However, software implementation can require greater amounts of power. Pure software approaches that lack sufficient power can result in processing delays which, in the context of image processing, can lead to unacceptable performance degradation, such as dropped frames, reduced resolution, dropped pixels, among others. Techniques can reduce power requirements for maintaining real time processing by using one or more hardware elements, such as a microcontroller coupled to one or more memory management units. The proposed techniques can be more robust against malicious software attacks, e.g., compared to software-based implementations, e.g., through the use of one or more hardware elements.

[0012] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of assigning a first virtual machine (VM) to a first image capture device and a second VM to a second image capture device , wherein each of the first and second VMs execute on a computing device communicably coupled to the first and second image capture devices and that includes a device memory and a plurality of image signal processor (ISP) blocks; allocating a first portion of the device memory to the first VM and a second portion of the device memory to the second VM; storing, by the first VM, processing parameters of a first ISP block in the first portion of the device memory; providing, by the first VM, data indicating a location of the first portion of the device memory' to a microcontroller coupled to the plurality of ISP blocks; transmitting, by the microcontroller and based on the data indicating the location of the first portion of the device memory, the processing parameters from the first portion of the device PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0013] Attorney Docket No. 56113-0734WO1

[0014] Client Ref: GP-307788-00-PCT memory to ISP registers corresponding to the first ISP block; and processing, by the first ISP block, image data from the first image capture device using the processing parameters. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0015] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. In particular, one embodiment includes all the following features in combination. Feature 1 : Providing the data indicating the location of the first portion of the device memory to the microcontroller comprises: writing a command to a first mailbox of the microcontroller assigned to the first VM and inaccessible to the second VM. Feature 2: Writing the command comprises: writing a command configured to generate an interrupt signal. Feature 3: Transmitting the processing parameters from the first portion of the device memory occurs in response to generating the interrupt signal. Feature 4: Actions include loading, by the microcontroller, the processing parameters from the first portion of the device memory; and writing, by the microcontroller, a command to a first mailbox of the microcontroller assigned to the first VM and inaccessible to the second VM upon completion of transmitting the processing parameters from the first portion of the device memory to the ISP registers corresponding to the first ISP block. Feature 5: Actions include assigning, by the microcontroller, a (i) first identifier to the first image capture device and (ii) a second identifier to the second image capture device; performing one or more of the following using a memory management unit and the assigned first and second identifiers: blocking the second VM from accessing the processing parameters stored in the first portion of the device memory allocated to the first VM; blocking the second VM from accessing a mailbox of the microcontroller allocated to the first VM; or blocking the second VM from accessing the image data from the first image capture device stored in the first portion of the device memory allocated to the first VM. Feature 6: Transmitting the processing parameters from the first portion of the device memory comprises: providing an identifier, with a request for the processing parameters of the first ISP block, to a memory management unit. Feature 7: Allocating the first portion of the device memory to the first VM comprises: allocating (i) a raw data portion of the device memory to the first VM to store the image data from the first camera and (ii) a processed data portion of the device memory to the first VM to store the image data processed by the plurality one or more ISP blocks. Feature 8: Actions include storing raw image data from the first image capture device within the allocated raw PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0016] Attorney Docket No. 56113-0734WO1

[0017] Client Ref: GP-307788-00-PCT data portion of the device memory; loading, by the microcontroller, the raw image data from the allocated raw data portion of the device memory; and providing the loaded raw image data as the image data to the first ISP block prior to the image data processing by the first ISP block. Feature 9: Storing the processing parameters of the first ISP block in the

[0018] 5 first portion of the device memory comprises: storing data indicating one or more of exposure control, white balance correction, auto-focus, noise reduction, sharpness improvement, color correction, or gamma correction. Feature 10: Actions include processing, by a second ISP block, second image data after processing the image data by the first ISP block. Feature 11: Actions include processing, by a second ISP block, second

[0019] 10 image data in parallel with processing the image data by the first ISP block. Feature 12: Actions include generating a first identifier corresponding to the first camera and a second identifier corresponding to the second camera. Feature 13: The first and second image capture devices are on the device and communicably coupled to the device. Feature 14: The first and second image capture devices are on another device different than the device, and wherein the first and second image capture devices of the other device are communicably coupled to the device.

[0020] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG. 1 shows an example electronic device.

[0023] FIG. 2 shows another example device.

[0024] FIG. 3 shows an example process for ISP virtualization.

[0025] FIG. 4 is a diagram illustrating an example of a computing system used for ISP

[0026] 25 virtualization.

[0027] Like reference numbers and designations in the various drawings indicate like elements.

[0028] DETAILED DESCRIPTION

[0029] This specification describes processes and systems for the virtualization of Image Signal Processor Hardware (ISP HW) provided in a computing device (e.g., an edge device

[0030] 30 such as a smartphone). In general, an ISP processes image data from one or more cameras, PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0031] Attorney Docket No. 56113-0734WO1

[0032] Client Ref: GP-307788-00-PCT e.g., on a smartphone. Many modem systems, such as a smartphone, have multiple cameras. In the proposed systems, each camera of a device can, in some cases, be associated with a different virtual machine (VM). Each VM can access HW as if the VM owns dedicated HW. A VM can host one or more programs, where each program has access to HW resources controlled by the VM. A VM can handle coordination with ISP blocks through one or more elements, such as a microcontroller.

[0033] In some cases, each VM has exclusive access to data obtained from a corresponding camera. For example, a VM configured to process data from a first camera can be the only VM, of a multiple VMs handling multiple camera data feeds, that can access data from the first camera.

[0034] The proposed techniques include VM processing in both parallel and time-shared environments. For example, the described systems can include VMs that process data from one or more cameras in parallel or one data set at a time (such as image frame by image frame or a portion of an image frame by another portion). In general, parallel processing can include processing one or more images at least partially simultaneously, e.g., using one or more hardware elements of a graphics processor unit, central processing unit, or system on a chip. In general, time-shared processing can include processing one or more images obtained from memory and, after processing the one or more images, obtaining new data for subsequent processing. In some cases, time shared processing of subsequent data begins only after previously processed data output is written to memory.

[0035] Techniques described can enable real time processing of data, e.g., using one or more VMs. Real time processing can include processing with latency below a threshold latency time, such as ten milliseconds, for processing data. Where processing blocks are shared, e.g., to serially process data from one or more sources, latency tolerance may be higher than when multiple processing bocks are configured to concurrently process data.

[0036] Techniques can include a processing device from dropping frames or dropping pixels when processing multiple image streams, such as one image stream from a front facing camera while simultaneously processing an image stream from a rear facing camera (or from simultaneous processing an image captured by each rear-facing camera). In some cases, different image streams can be processed using one or more VMs. In some cases, similar streams of image data can be processed using one or more VMs, e.g., to determine authenticity of an image stream by separately process cameras capturing similar views (such as views from a rear side of a device), to generate different output using the similar image streams (such as using generative artificial intelligence to generate first output from PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0037] Attorney Docket No. 56113-0734WO1

[0038] Client Ref: GP-307788-00-PCT a first similar image stream and different second output from a second similar image stream where the first and second similar image stream are similar to one another, or other processes to generate different output).

[0039] In some cases, ensuring real time processing includes performing virtualization using one or more hardware elements. For example, software methods to process and route data could introduce processing delay (e.g., leading to frame or pixel drops) without the inclusion of power heavy software processing overhead which is especially costly on batten- operated devices, such as smartphones. Such processing delays, while potentially less problematic for virtualization of CPU or other processing, can cause serious deterioration in image processing, e.g., through frame or pixel drops.

[0040] In some cases, delay in processing can deteriorate camera image processing significantly, e.g., due to lag between an image being taken and a result image being provided for display. Delay can cause pixels to be dropped, e.g., causing image quality deterioration. A frame may be thrown away a threshold number of pixels are dropped. A frame may be dropped due to frame buffer over flow. Such drops may appear as choppiness to a viewer. Techniques described can isolate a timing critical part of processing from a VM's operation (which can be much more prone to delay than hardware or microcontroller). A timing critical part of processing can include providing ISP parameters. A VM can send ISP parameters to a system periodically over time. For example, a VM can send ISP parameters every set period of time, e.g., 30 milliseconds, or every frame. By using hardware to receive and process the data based on the VM provided ISP parameter, the system can reduce potential delays, e.g., with direct processing of data by the VM.

[0041] Techniques described in this document can enable increased security of image signal processing. For example, these techniques can increase confidentiality and integrity of data. Data for processing by a first VM can remain separate from data for processing by a second VM different that the first VM. By separating the data, information leaking from one VM to another can be prevented. Information leak can have negative effects, e.g., if a system includes a video playback device, a leak can lead to unauthorized copying of corresponding video playback. If a system includes an image capture device, an input source can be a secure camera, e.g., used for biometric authentication, such as finger print detection, iris detection, among others. An attacker can use information leaks for malicious purposes. Information leaks can cause privacy violation issues on user devices.

[0042] Techniques can help to prevent injection of information from one VM to another, e.g., in cases where one VM attempts to inject data into a memory used by another VM. In PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0043] Attorney Docket No. 56113-0734WO1

[0044] Client Ref: GP-307788-00-PCT general, techniques can help to prevent malicious operations on a device, e.g., an application that may be a trojan horse, a bad script on a web page, among other malicious operations. Malicious operations can result in, e.g., unauthorized log ins, imposters, fabricating records, among other effects. Techniques can help to prevent one VM from corrupting data of another VM, e.g., by ensuring that a VM can only access data associated with itself and not with data associated with other VMs. Techniques can include data corruption prevention, such as separating a control of a given MV from other VMs, e.g., to prevent denial of service type of attack. If one bad VM tries to disrupt another VM’s processing by taking over control (e.g. distorting image data, making color completely black), robust virtualization can prevent it. Techniques can include prevention by filtering DMA read and write access to ISP parameters on DRAM.

[0045] In some cases, a set of one or more VMs process data from one or more cameras using only one set of ISP blocks. For example, ISP blocks can be included in a system on a chip architecture. ISP blocks can include one or more hardware processing circuits designed to process image signals. For example, an ISP block can include a processing circuit to reduce noise in an image, correct colors, sharpen, correct gamma, or perform other processing, to name just a few examples.

[0046] Techniques described in this document can also enable separate VMs being used to process separate data streams obtained from separate cameras. Techniques can be performed using software or hardware elements. For example, software (e.g., running on a VM host processor(s)) can implement a hypervisor, e.g., to divide or allocate image data streams to ISP processing elements. In some cases, such as in mobile devices where power constraints are high or in cases where image buffer will result in dropped frames or pixels if processing delay occurs, software overhead from such a hypervisor (e.g., from processes of sanitization of ISP control commands or serialization of ISP resource usage) cancan be non-optimal.

[0047] In some cases, hardware elements can be used to perform techniques described to help reduce software overhead. For example, a microcontroller can be implemented within the ISP HW. A microcontroller can be used to provide one or more microcontroller mailboxes for each VM of one or more VMs in use. The microcontroller can have write and read access to ISP registers. The microcontroller can receive interrupts from ISP blocks. The microcontroller can include direct memory access module (DMA), e.g., used by the microcontroller to program ISP. In some cases, the DMA can read from dynamic random-access memory (DRAM) and write to an ISP register, e.g., with memory PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0048] Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT management unit (MMU) or microprocessor unit (MPU) protection. In some cases, DMA is controlled only by the microcontroller, e.g., and not by the host processor of the one or more VMs. In some cases, a sequence identifier (SID) is attached to both DRAM read and ISP register write. In some cases, the DMA can include a SID-based firewall for ISP register access, e.g., where ISP register access is confirmed or denied by the firewall based on a provided SID. Unique SID values can be associated with each ISP register.

[0049] Techniques can include one or more processors to perform image processing. ISP blocks of the ISP HW can include read and write DMAs that have an input-output memory management unit (I0MMU), e.g., with SID support. ISP blocks of the ISP HW or processors, such as processors of a VM host, can be configured to apply a consistent SID that is associated with a given camera data stream.

[0050] In some cases, a first process configures an MMU or MPU, e.g., by changing one or more parameters of the MMU or MPU. MMU can refer to an I0MMU, e.g., used by devices such as ISP devices, DMA devices, or controllers. MMU can refer to an MMU used by a CPU, such as the CPU 112 of FIG. 1. Parameters to be changed, e.g., by a first process, can include MMU page table. MMU page table can be generated to map a virtual address (VA) to a physical address (PA). MMU can select a table for each SID. A VA can refer to virtual memory addressing rather than virtualization in a VM. By setting a table for each SID, MMU allows one VM (e.g., with a specific SID) to access only an allowed PA range. MMU can prevent a VM from accessing a PA range that is not associated with the given VM. MPU can give access to certain address range based on SID. MPU can perform blocking. In some cases, the MPU performs blocking using a given SID.

[0051] In some cases, the first process can be a type of hypervisor process. The MMU or MPU can be configured to prohibit direct access from a VM to an ISP register. ISP registers can inform processing to be performed by the ISP, e.g., using one or more processing blocks configured to perform operations of the ISP. The first process can include assigning one or more cameras of one or more devices to one or more respective VMs. For example, for a device with a front facing camera and a rear facing camera, the first process can assign the front facing camera to a first VM and the rear facing camera to a second VM. The first process can assign mailboxes of a microcontroller to each of the assigned VMs. For example, the mailboxes can be used by VMs to write or read data from the microcontroller, e.g., to read or right data from one or more ISP registers. The first process can assign one or more image buffers, e.g., on DRAM, to one or more VMs. In some cases, the MMU, MPU, I0MMU, or a combination of these among others, can enforce access, e.g., to ensure PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0052] Attorney Docket No. 56113-0734WO1

[0053] Client Ref: GP-307788-00-PCT that each VM can access only the buffer allocated to the VM. The first process can include setting up an I0MMU of the ISP to ensure that data from a first camera associated with a first VM is written to a memory buffer that only the first VM has access to, e.g., via the assignments of image buffers to one or more VMs. For example, the I0MMU can assign a unique SID to each camera stream and maintain an SID table to ensure that camera streams are stored in memory locations specific to that SID which can correspond to both one or more cameras and one or more VMs assigned to process data from the one or more cameras.

[0054] The first process can include allocating an ISP control buffer, e.g., on DRAM, to each VM. The ISP control buffer can include ISP register contents, such as image processing parameters or read or write DMA addresses. In some cases, the ISP control buffer is only accessible from the VM that is allocated to that buffer.

[0055] The first process can include configuring a microcontroller’s I0MMU. For example, configuring can include configuring the I0MMU so that DMA can only read from the buffer corresponding to the SID programmed by the microcontroller. Configuring the I0MMU can help ensure that the DMA accesses only the necessary VM data. Configuring can include adjusting one or more parameters of the I0MMU. Configuring can include changing one or more parameters, such as page tables in DRAM that describe a mapping from a VA to PA. A hypervisor can store page tables in DRAM. An MMU can access the stored tables. The MMU can read the tables from memory when necessary, e.g., to prevent unauthorized access based on SID.

[0056] In some cases, techniques can include using a firewall to restrict access to register regions. A firewall can include an access control mechanism to prevent memory access and can be configured using any appropriate access protection technique. For example, a microcontroller can set up a firewall for a DMA so that the DMA can write to the target register region, e.g., a region assigned for each SID. The region for each SID can be assigned by a microcontroller.

[0057] Techniques can include a camera control process. For example, a camera control process can include one or more VMs writing ISP control parameters, such as register values, to a shared buffer. The process can include one or more VMs transmitting data to a microcontroller to request a location of the parameters through a mailbox of the microcontroller. The process can include a microcontroller receiving such a request from the VM, e.g., requesting a location of parameters. The process can include a microcontroller loading ISP register payload from memory and wnting the ISP registers PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0058] Attorney Docket No. 56113-0734WO1

[0059] Client Ref: GP-307788-00-PCT using DMA. The process can include the ISP processing input data from one or more cameras, generating output data from the processing, and writing the output data to a buffer allocated for one or more specific VMs. The process can include transmitting information to the one or more VMs, e.g., that processing of image data is complete. Transmitting information can include using one or more mailboxes of a microcontroller. Transmitting information can include one or more commands, such as specify ISP parameter buffer addresses, start ISP parameter upload, ISP Start, ISP Stop, or a combination of these among others. In general, mailboxes of a microcontroller can be used to transfer data between two processes. A first process can write data to the mailbox, which can be a memory location within stored memory. The first process can trigger an intermpt which can cause the microcontroller to send data to a second process. In response to receiving the data of the interrupt, the second process can read the data from the mailbox, write an exit status, and trigger an interrupt to be sent to the first process. In a similar manner, the first process can read the exit status at the mailbox in response to receiving an interrupt.

[0060] In some cases, techniques include providing independent and secure control for each VM. For example, each mailbox of a microcontroller can be protected. A VM can issue a command to a microcontroller through using a mailbox of the microcontroller. In some cases, the mailbox is only accessible via the given VM, e.g., and no other VM. In some cases, ISP parameters are protected. For example, a VM can store ISP parameters into memory space allocated to the given VM. A system, as described, can be configured to prohibit other VMs from accessing this memory space. A microcontroller can assign an SID to a DMA of the Microcontroller. The DMA can read the ISP parameters from the buffer allocated to the given VM. In some cases, ISP registers are protected. For example, in parallel processing ISP blocks, a microcontroller DMA can only write to ISP registers that a target VM is associated with, e.g., because the system can include an SID-based firewall. For a time-shared ISP block, a microcontroller can issue DMA requests one at a time, e.g., to prevent mixing commands.

[0061] In some cases, input and output data are separated. For example, an I0MMU of an ISP can determine whether or not camera data is written to a buffer owned by a target VM. Each camera providing data to a processing device can be associated with a camera stream. Each camera stream can be associated with a unique SID. The SID can be assigned by ISP HW. The I0MMU can be programmed to block access from a VM to other memory space allocated to other VMs. If a VM tries to send camera data with which the VM is not associated, to the VM’s own buffer, I0MMU can detect this and block it by checking and PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0062] Attorney Docket No. 56113-0734WO1

[0063] Client Ref: GP-307788-00-PCT determining that the SID doesn’t match the target address, e.g., using one or more comparator circuits.

[0064] In some cases, an I0MMU of an ISP determines that camera data is read from a buffer allocated to a particular target VM. For example, a microcontroller can set a write SID to ISP HW’s read. Therefore, ISP HW can only read from a buffer allocated to a given VM. ISP HW can check if the SID of a read and write is the same when read DMA is used as input. The microcontroller can check the SID to ensure that each VM programmed the right SID to ISP, e.g., by comparing the programmed value to a table of known SID values, such as values assigned by the microcontroller. If a VM tries to write to memory space allocated to another VM, an I0MMU can block the write action in response to the I0MMU comparing the stream ID (e.g., including an immutable SID) to a target address and determining that the SID does not match the target address, e.g., the target address of the write action does not correspond with an SID provided with the write action. In some cases, the I0MMU can use a table of SID values and target addresses to ensure that writing is limited to specific VMs. If a VM tries to write to an address of another VM using ISP or DMA, an I0MMU can be used to block the action. If a VM tries to write to an address of another VM as a CPU action, an MMU can be used to block the action. In some cases, I0MMU can refer to instances of an MMU.

[0065] The processing overhead of the proposed techniques can be reduced, causing a technical improvement, for one or more of the following reasons. Data protection can be performed using hardware protecting elements, such as a microcontroller or I0MMU. Protection of data to and from the ISP, including high bandwidth image data, can be done via hardware elements. Because of the hardware implementation, software overhead of digitally sanitizing data, e.g., to prevent mixing of signals or processing between different camera streams, can be eliminated or reduced. Although purely software implementations could be achieved, such implementations would require more power to maintain a threshold level of processing delay, a threshold which can be especially low in image processing. The hardware approach described can be useful in cases where power restraints limit the available power or in cases where energy conservation is important, such as on mobile devices or devices running on limited battery supplies.

[0066] FIG. 1 shows an example electronic device 100 that includes a system-on-chip (SoC) 106. The device 100 can be any device of the devices 102, such as a smartphone 102a, a tablet 102b, a laptop 102c, or a wearable device 102d. In general, the device 100 can be any device that processes data. The device 100 includes the SoC 106 and memory PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0067] Attorney Docket No. 56113-0734WO1

[0068] Client Ref: GP-307788-00-PCT devices 104. The SoC 106 can transmit data to and from the memory device 104 to perform operations. The SoC 106 includes processing blocks 108, a memory unit 110, and a central processing unit (CPU) 112. The processing blocks 108 can be configured to perform operations, such as speech detection, machine learning processing, telecommunication signal processing, or a combination of these, among others. The processing blocks 108 can read or write to the memory devices 104. The memory devices 104 can include various memory elements, such as dynamic random-access memory (DRAM). The processing blocks 108 can include ISP blocks for image signal processing. The memory unit 110 can be used as an interface for reading or writing data to the memory devices 104. The memory unit 110 can store data for processing or processing results generated by processing in the SoC 106. The CPU 112 can execute instructions, perform calculations, or help to control system functionality. The CPU 112 can include a single processor or multiple cores. The CPU 112 can be configured for general-purpose computing. The processing blocks 108 can be used to perform one or more functions not performed by the CPU 112 — e.g., processes that are performed frequently enough, or are process intensive enough, to warrant dedicated circuitry.

[0069] The device 100 performs processing, in the example of FIG. 1, with raw image data 120. In some cases, the raw image data is obtained using one or more cameras, such as raw image data obtained from a single device with multiple cameras or multiple devices with one or more cameras. The processing blocks 108 can perform operations on the raw image data 120. For example, one or more ISP blocks of the processing blocks 108 can perform operations on the raw image data 120. The described techniques of virtualization can be used by the device 100, e.g., for processing obtained image data, such as the raw image data 120. In some cases, virtualization can be used to isolate processing of data feeds from different cameras and perform parallel or time-wise sharing of processing from that data. In some cases, the device 100 includes an image processing engine, e.g., performed by one or more elements shown in FIG. 1, that processes obtained image data, such as the raw image data 120.

[0070] FIG. 2 shows another example device 200. In some cases, the device 200 is an example of the device 100 of FIG. 1. The device 200 includes a CPU 208, a memory unit 210, and processing blocks 216. Elements of the device 200 can be example of the device 100 shown in FIG. 1, such as elements of the SoC 106 or the memory devices 104. For example, the CPU 208 can be an example of the CPU 112 of FIG. 1. The memory unit 210 can be an example of the memory unit 110 of FIG. 1. The processing blocks 216 can be an PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0071] Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT example of the processing blocks 108. In general, the device 200 can obtain camera image data 204, process the camera image data 204, and generate processed data 230.

[0072] In some cases, the device 200 includes camera sensors 202. In some cases, the camera sensors 202 are included in other devices that are communicably connected to the device 200. The cameras 202 can include two or more cameras where the device 200 includes one camera and another device communicably connected to the device 200 includes another camera. The camera sensors 202 can include any type of sensor, such as infrared sensor, light detection and ranging (LIDAR) sensor, visual sensor, or a combination of these, among others.

[0073] The camera image data 204 can include image data from one or more sensors, e.g., camera 1 image data 204a from a first camera and camera N image data 204b from an Nth camera where N can be any number greater than one. At least a subset of the N cameras can be included in the device 200. In some cases, devices with camera sensors providing the image data 204 can include a smartphone, laptop, tablet, or other device. In some cases, the camera image data 204 is obtained from multiple devices, e.g., a first smartphone and a second smartphone.

[0074] The CPU 208 can include one or more processing circuits configured to execute instructions to perform operations for a virtual machine. In some cases, a VM hosts a computing process, such as a program that includes displaying or processing images captured by a camera, e.g., for a web-based video call or for a face detection scan for device security. A program executing on the VM can read data that has been processed by HW ISP blocks.

[0075] In some cases, the CPU 208 performs one or more software operations to setup image processing virtualization. For example, the CPU 208 can perform operations of a hypervisor. A hypervisor can include instructions that when executed manage virtualization operations, including setting up and managing various VMs.

[0076] The CPU 208 can include a setup program that assigns one or more cameras to one or more VMs. The CPU 208 can instantiate one or more VMs if one or more VMs are not already instantiated. The CPU 208 can assign one or more mailboxes to each of the VMs. For example, in FIG. 2, the CPU 208 can assign camera 1 to VM-1 208a and can assign mailbox-1 (MB-1) 218a to the VM-1 208a. In some cases, a single VM can process data from multiple cameras, multiple VMs can process data from a single camera, and / or a single VM can process data from a single camera assigned thereto. In some cases, each MB is only accessible from by the assigned VM. For example, an MMU or MPU can be PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0077] Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT used to enforce exclusive read or write operations for particular VMs to particular assigned MBs.

[0078] The CPU 208 can assign one or more image buffers, e.g., on the memory unit 210, to one or more VMs. The memory' unit 210 can, in some cases, be DRAM. VM-K represents at least one additional VM corresponding to a Kth VM where K can be any number greater than one. Similarly, MB-J represents at least one additional MB corresponding to a Jth MB where J can be any number greater than one.

[0079] In some cases, the VM-1 208a writes ISP control parameters such as register values to a shared buffer. The ISP control parameters can indicate parameters for controlling how processing is performed for a given camera stream, such as one or more parameters for one or more of the following: exposure control, white balance correction, auto-focus, noise reduction, sharpness improvement, color correction, gamma correction, or a combination of these among others.

[0080] The VM-1 208a can write ISP control parameters to an ISP parameter 212 portion of the memory unit 210. In some cases, the portion of the memory unit 210 can be exclusively accessible by the VM-1 208a, e.g., enforced by an MMU or MPU. For example, an MMU can be used in the device 200 to manage memory' access. The MMU can check whether an identity of a VM accessing the portion of the memory unit 210 matches an authorized accessor, e.g., a VM from one or more VMs hosted by the CPU 208. If the MMU confirms the identity matches an identify of an authorized accessor, the MMU can allow the access. If not, the MMU can block the access, e.g., the writing of ISP parameters.

[0081] The VM-1 208a can write data to MB-1 218a. For example, the VM-1 208a can write data to MB-1 218a that indicates a location of the portion of the memory unit 210 used by the VM-1 208a to store ISP control parameters. Read or write access can be controlled for the MBs similar to the memory unit 210. For example, an MMU can be used in the device 200 to check an identify of a VM accessing the MB-1 218a. If the MMU determines the accessor identify matches a previously stored authorized accessor, the MMU can allow the access. If not, the MMU can block the access, e.g., the writing of data to the MB-1 218a. Data that indicates a location can include a memory storage address indication the portion of the memory unit 210.

[0082] The processing blocks 216 include a controller 218, which reads data from one or more MBs. For example, the controller 218 can read data from the MB-1 218a that indicates the location of the portion of the memory unit 210 used by the VM-1 208a to store PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0083] Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT

[0084] ISP control parameters. The controller 218 can use the read data to load data from the portion of the memory unit 210 indicating ISP parameters. In some cases, a firewall is used, e.g., to help ensure the controller 218 accesses ISP parameters for specific VMs. The firewall can be based on identifiers or address ranges.

[0085] In some cases, a firewall is implemented in the processing blocks 216 as one or more hardware elements. A DMA generate one or more SID values for one or more image streams using a register. A microcontroller can assign an SID that corresponds to a target VM by writing a value to this register. When DMA accesses a DRAM or ISP register, DMA can send a command with an SID as embedded information. The firewall can include a table mapping SID to an allowed address range. The firewall can check the SID of a coming stream from DMA. The firewall can allow access to the target if the address is within an allowed range. The firewall can help prevent one or more malicious attacks, such as an attack that includes: a compromised VM-1 that writes a payload to the DMA and places it in VM-1 ’s DRAM address range; where the payload includes a write command to VM-2’s ISP register; a microcontroller starts the DMA with SID for VM-1; DMA tries to write to VM-2’s ISP address. The techniques described can prevent this attack because the described firewall can prevent the DMA attempt to write in VM-2’s ISP address.

[0086] In some cases, an I0MMU can be used in place of a firewall and vice versa. In the described techniques, an I0MMU can be used at the DRAM access side (e.g., to control access to DRAM addresses). The I0MMU can be a preexisting part of a system. If a system is not equipped with an I0MMU, a firewall can be used to control DRAM access. A firewall can be used at the register side (e.g., to control access to ISP registers). A firewall can be more lightweight compared to an I0MMU. I0MMU can do more than just filtering access based on address, SID, or a combination of both, among other data. An I0MMU can perform VA to PA mapping and other types of processing and can consume significant resources and be a large physical element. The techniques propose using a firewall, as discussed, to control access to ISP registers as a lightweight implementation to perform filtering on access requests.

[0087] In some cases, a hypervisor or microcontroller can check a DMA payload. An entity can check the payload to make sure a payload includes only benign commands, e.g., according to a list of one or more non-benign or benign commands. If no non-benign commands are detected, or all commands are benign, an entity can determine that I0MMU or a firewall are not needed. However, in instances where such processing by an entity, such as a hypervisor or microcontroller, requires significant power or time, include an PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0088] Attorney Docket No. 56113-0734WO1

[0089] Client Ref: GP-307788-00-PCT

[0090] I0MMU or firewall to help ensure exclusive access can be more optimal as it may be less likely to cause processing latency. The controller 218 can provide the loaded data to ISP blocks 220. The ISP blocks 220 can include one or more hardware circuits configured to process image data, such as one or more application specific integrated circuits (ASIC).

[0091] In some cases, the ISP blocks 220 use loaded data provided by the controller 218 to process image data. For example, the ISP blocks 220 can obtain data from the memory unit 210 allocated for image data 214, such as ISP control parameters. The memory portion allocated for the image data 214 can be separated into separate portions for different VMs. For example, for processing using the ISP parameters provided by the VM-1 208a, the ISP blocks 220 can obtain data within a portion of the image data 214 allocated to the VM-1 208a. Camera data can be sent directly to memory (e.g., DRAM through the I0MMU). The ISP blocks 220 can obtain the data at the DRAM via I0MMU as long as the memory address of the DRAM corresponds to the ISP parameters being used by the ISP blocks 220 for processing. The I0MMU can block accesses for memory addresses not assigned to a corresponding provided SID.

[0092] In some cases, a camera driver is used to assign specific cameras to specific identifiers. For example, the controller 218 or the CPU 208 can perform operations of a camera driver to assign a first camera that generates the camera 1 image data 204 a first identifier and an Nth camera that generates the camera N image data 204b an Nth identifier. An example assignment table is shown in item 209 of FIG. 2. In some cases, the camera driver uses the generated table 209 for configuring input and output of image data, e.g., using an I0MMU element to control reading and writing of image data to the memory unit 210. In some cases, the assignment table can be generated by a CPU, e.g., by a hypervisor operating on a CPU.

[0093] In some cases, an I0MMU is used to prevent the ISP blocks 220 from obtaining image data for processing using parameters from the VM-1 208a from a camera that is not associated with the VM-1 208a. In some cases, an entity other than an I0MMU is used. For example, another entity can be configured to block access attempts based on SID and address, such as an entity similar to the described firewall. Additional firewalls can be used, e.g., in devices that do not include an I0MMU.

[0094] The I0MMU can include a table of values including one or more identifiers, such as the table 209. Each identifier of the identifiers can correspond to a particular camera, e.g., of the N cameras used to obtain the camera image data 204. An identifier can include a sequence identifier (SID). A sequence identifier can be any value, such as numbers or PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0095] Attorney Docket No. 56113-0734WO1

[0096] Client Ref: GP-307788-00-PCT letters. Example identifiers are shown in table 209. Other identifiers, such as one, two, three, A, B, C, or a combination of these can be used. In some cases, the controller 218 generates an identifier for each camera configured for processing, e.g., a first camera is assigned an SID of “1” and a second camera is assigned an SID of “2”. When the ISP blocks 220 have loaded ISP parameters provided to the memory unit 210 by the VM-1 208a, RDMA of the ISP blocks 220 can obtain data from the image data 214 associated with the VM-1 208a. An I0MMU can compare a requested memory location and a provided identifier to determine if the identifier corresponds to the memory location or if the identifier corresponds to a different memory location. An identifier can be provided to an I0MMU as part of a memory access command.

[0097] In some cases, the identifier is immutable. For example, when an RDMA reads data, the identifier can be specified by a controller, such as the controller 218. The processing blocks 216 can use the same identifier for RDMA and WDMA. If input is an outside source, such as an image sensor, a controller can specify an identifier to the blocks 216. One or more hardware elements can be used to help ensure an identifier set by a controller is not modified. The identifier can help ensure image data and ISP parameters are only read from and written to an address range owned by a target VM. In some cases, a VM can only access its portion of ISP registers through DMA. In some cases, the identifier is not changed by software on a VM. In some cases, hardware blocks, such as the processing blocks 216, do not change an identifier once generated.

[0098] In some cases, the ISP blocks 220 write output data to the memory unit 210. For example, a WDMA element of the ISP blocks 220 can write output data to a portion of the memory unit 210 allocated for the image data 214 and specifically allocated to the VM-1 208a. In some cases, each VM can be assigned both a read and write memory location for image data. For example, a read memory location can store raw data from an assigned camera and a write memory location can store processed data generated by the ISP blocks 220.

[0099] As discussed, a VM can host any ty pe of process, such as a process for displaying camera footage on a device display. A VM can for example host a video calling process where camera footage is obtained by a device and displayed on a self-view in a calling interface. In the example of FIG. 2, the VM-1 208a can obtain data from an output location of the memory unit 210. An access control unit, such as an MMU, can be used to ensure that only VMs associated with particular image data are able to access the data in the memory unit 210. For example, an MMU can check an identifier of the VM-1 208a to PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0100] Attorney Docket No. 56113-0734WO1

[0101] Client Ref: GP-307788-00-PCT ensure that the location of memory requested for access by the VM-1 208a is authorized for access by the VM-1 208a. After accessing the data, the VM-1 208a can provide the processed data 230 for a process being performed by the VM-1 208a, such as displaying the processed data 230 or running subsequent processing on the processed data 230, such as security algorithms for facial detection. For example, the VM-1 208a can be an application operating on the device 200200. The processed data 230 can include a camera feed from a first camera sensor of the device 200200 or another device. The processing of multiple camera feeds on the device 200 can remain distinct. This can be especially important when simultaneously capturing data from multiple cameras. For example, using a front facing camera to securely unlock a device might require different ISP parameters, such as increased brightness or contrast, compared to capturing data with a back facing camera for a current ongoing video call.

[0102] In some cases, techniques can include using the same camera data to generate different output data, e.g., using two different processing pipelines. Processing can be managed by one or more VMs. An example scenario would be using a front facing camera in video call and using a same image feed for security authentication. The security authentication might require different image processing, e.g., comparisons to secure template user, a higher brightness processing, or other control parameter. A single camera sensor can, e.g., provide data for processing by two or more VMs of the CPU 208.

[0103] In some cases, one or more VMs store ISP parameters as DMA commands in DRAM. Each MB of the controller 218 may be limited in space, e.g., tens of bytes. By iteratively providing ISP parameters, a VM can effectively increase the available ISP parameters able to be handled by a given controller because the VM only provides a subset of all stored commands for ISP parameters at a time. A MB of a controller, such as the controller 218, can be used by a VM to inform the controller about a location of ISP parameters. The controller can start a DMA to program an ISP, e.g., to inform processing of data by the ISP blocks 220.

[0104] FIG. 3 is a flowchart of an example process 300 for ISP virtualization. For convenience, the process 300 will be described as being performed by a system of one or more computers, located in one or more locations, and programmed appropriately in accordance with this specification. For example, the device 200 of FIG. 2 or the device 100 of FIG. 1, appropriately programmed, can perform the process 300.

[0105] The process 300 includes assigning a first virtual machine (VM) to a first image capture device and a second VM to a second image capture device (302). For example, the PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0106] Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT

[0107] CPU 208 or the controller 218 can assign a first camera to the VM-1 208a. An Nth camera, corresponding to the camera N image data 204b, can be assigned to the VM-1 208a or another VM. In some cases, each of the first and second VMs execute on a computing device communicably coupled to the first and second image capture devices and that includes a device memory and a plurality of ISP blocks.

[0108] The process 300 includes allocating a first portion of device memory to the first VM and a second portion of the device memory to the second VM (304). For example, the CPU 208 can allocate a first portion of the memory unit 210 to the VM-1 208a. Another portion of the memory unit 210 can be allocated to another VM, such as the VM-K. In some cases, the controller 218 allocates memory to one or more VMs.

[0109] The process 300 includes storing processing parameters of aa first ISP block in the first portion of the device memory (306). For example, the VM-1 208a can store processing parameters in the ISP parameter 212 portion of the memory unit 210.

[0110] The process 300 includes providing data indicating a location of the first portion of the device memory to a microcontroller coupled to the plurality of ISP blocks (308). For example, the VM-1 208a can store data at MB-1 218a indicating a location where the VM- 1 208a stored processing parameters.

[0111] The process 300 includes transmitting the processing parameters from the first portion of the device memory to the first ISP block (310). For example, the controller 218 can load a portion of data from the ISP parameter 212. The controller 218 can provide the loaded data to the ISP blocks 220, e g., to ISP registers corresponding to the first ISP block.

[0112] The process 300 includes processing, by the first ISP block, image data from the first image capture device using the processing parameters (312). For example, the controller 218 can provide data from image data 214 to the ISP blocks 220 for processing. For processing using processing parameters provided by the VM-1 208a, the ISP blocks can be provided image data from a first camera that is assigned to the VM-1 208a. In some cases, memory management units can be used to ensure that processing parameters are used for processing the correctly assigned image data, e.g., that processing parameters for processing data from the first camera are actually used in the ISP for processing data from the first camera. In some cases, an identifier is added to data corresponding to raw or processed data or processing parameters for processing data from a first camera. For example, all data related to processing data from a first camera can include an identifier that identifies the first camera, such as a SID. PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0113] Attorney Docket No. 56113-0734WO1

[0114] Client Ref: GP-307788-00-PCT

[0115] In some cases, the process 300 includes allocating (i) a raw data portion of the device memory to the first VM to store the image data from the first camera and (ii) a processed data portion of the device memory' to the first VM to store the image data processed by the plurality one or more ISP blocks. In some cases, the process 300 includes: storing raw image data from the first image capture device within the allocated raw data portion of the device memory; loading, by the microcontroller, the raw image data from the allocated raw data portion of the device memory; and providing the loaded raw image data as the image data to the first ISP block prior to the image data processing by the first ISP block. For example, a combination of one or more elements can control where raw data is stored, e.g., where to store data in a buffer associated with a VM. A combination of elements can include a combination of a hypervisor, VM, and microcontroller. A hypervisor can allocate a certain physical address range to each VM by setting an MMU page table. This page table can map each VM’s VA to PA. Each VM can be limited to see only VA so can’t obtain data indicating addresses assigned to other VMs. When a VM accesses a certain address (VA), MMU can map it to a physical address within an allocated range. Software running on one VM can determine where to store RAW data within its allocated VA range. Software on a VM can inform a controller, such as the controller 218 of FIG. 1, of what address the controller and ISP should use for processing data associated with a given VM, such as addresses within the VM’s VA. A microcontroller and DMA can access memory using a VM’s VA and a VM’s SID. A MMU can translate the VA to PA. This PA can be within a range allocated to the VM.

[0116] In some cases, a first camera and a second camera configured to capture image data are included on a same electronic device. For example, the camera image data 204 can be captured by two different cameras of a same device. In some cases, the camera image data 204 can be captured using two different cameras of two different devices. In some cases, a single camera image stream is duplicated and processed separately using one or more of the techniques described in this specification.

[0117] In some cases, a first camera and second camera configured to capture image data are configured to point in different directions. For example, a first camera can be a frontfacing camera and a second camera can be a rear-facing camera. The front and rear-facing camera image data can be included in the camera image data 204.

[0118] In some cases, the process 300 includes providing processed image data for display on a device. For example, the processed data 230 can be displayed as part of a process performed by one or more of the VMs hosted by the CPU 208. PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0119] Attorney Docket No. 56113-0734WO1

[0120] Client Ref: GP-307788-00-PCT

[0121] In some cases, the process 300 includes assigning, by the microcontroller, a (i) first identifier to the first image capture device and (ii) a second identifier to the second image capture device; performing one or more of the following (e.g., using a memory management unit, firewall, or suitable process): blocking the second VM from accessing the processing parameters stored in the first portion of the device memory allocated to the first VM; blocking the second VM from accessing a mailbox of the microcontroller allocated to the first VM; or blocking the second VM from accessing the image data from the first image capture device stored in the first portion of the device memory allocated to the first VM. If a system does not have a memory management unit, such as an I0MMU, a firewall can be used, e.g., placed at a master port of DMA for memory access.

[0122] In some cases, the controller 218 loads parameters to an ISP block sequentially. For example, if only one DMA is available on a system, such as the device 200, for parameter upload, the one DMA can be called by the controller 218 sequentially for each set of parameters to be loaded. The controller 218 can include a microcontroller.

[0123] In some cases, the controller 218 loads new ISP parameters periodically, e.g., as image data is processed. For example, the controller 218 can load new ISP parameters for one or more ISP blocks once per frame being processed of image data. Once a frame can be, e.g., every 30 milliseconds or 16 milliseconds, among others.

[0124] In some cases, various actions described in reference to FIG. 2 occur in parallel. For example, two or more of the following actions can be performed at least partially in parallel: (i) a VM storing ISP parameters, (ii) a VM communicating with a controller indicating a location of the ISP parameters, (iii) the controller receiving one or more communications from VMs, (iv) image data being provided, e.g., by one or more hardware elements of ISP blocks or controller, to one or more ISP blocks, or (v) informing one or more elements by a controller, such as through MBs of the controller 218.

[0125] In some cases, MBs, e.g., used by the controller 218, include a set of shared buffers which can be accessed both from a host, such as the CPU 208, or client, such as the controller 218. Buffers can be mapped to addresses of the CPU 208. Buffers can be mapped to addresses of the controller 218. In some cases, the address space of the CPU 208 and the controller 218 are independent from one another. For example, each buffer corresponding to one or more MBs can have a corresponding address in both the CPU 208 and the controller 218. In some cases, MBs include a doorbell function. For example, a doorbell function can send an intermpt signal to a host or client when triggered. In some cases, host and client can poll a status of a MB to detect if there’s a command in the MB, PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0126] Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT e.g., instead of the doorbell function. The doorbell function can help to improve efficiency of operation.

[0127] In some cases, an information format is defined between a software of a host and software of a client, e.g., software of the CPU 208 and software of the controller 218. The content of a MB command can depend on the software architecture of the host and client. Possible commands can include one or more of the following: (i) Specify ISP Parameter Buffer Addresses, (ii) Start ISP Parameter Upload, (iii) ISP Start, (iv) ISP Stop.

[0128] MB operation can include a host writing information to a MB address. Writing the information can trigger a doorbell interrupt signal. A client can read the information and execute a task. The client can write an exit status to the MB. The writing can again trigger a doorbell intermpt signal. A host can read an exit status of the MB. Data of the MB can be stored in corresponding buffer addresses of the host and client, e.g., buffer addresses stored within the CPU 208 and the controller 218. In some cases, the CPU and controller can interact with locally stored buffers (with, e.g., software that communicates between a CPU and controller via an intermpt signal.

[0129] The order of operations in the process 300 described above is illustrative only, and can be performed in different orders in some cases. In some implementations, the process 300 can include additional operations, fewer operations, or some of the operations can be divided into multiple operations.

[0130] FIG. 4 is a diagram illustrating an example of a computing system used for ISP virtualization. The computing system includes computing device 400 and a mobile computing device 450 that can be used to implement the techniques described herein. For example, one or more components of the device 100 or 200 could be an example of the computing device 400 or the mobile computing device 450.

[0131] The computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device 450 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, mobile embedded radio systems, radio diagnostic computing devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

[0132] The computing device 400 includes a processor 402, a memory 404, a storage device 406, a high-speed interface 408 connecting to the memory 404 and multiple high- PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0133] Attorney Docket No. 56113-0734WO1

[0134] Client Ref: GP-307788-00-PCT speed expansion ports 410, and a low-speed interface 412 connecting to a low-speed expansion port 414 and the storage device 406. Each of the processor 402, the memory 404, the storage device 406, the high-speed interface 408, the high-speed expansion ports 410, and the low-speed interface 412, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 402 can process instructions for execution within the computing device 400, including instructions stored in the memory 404 or on the storage device 406 to display graphical information for a GUI on an external input / output device, such as a display 416 coupled to the high-speed interface 408. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. In addition, multiple computing devices may be connected, with each device providing portions of the operations (e g., as a server bank, a group of blade servers, or a multi-processor system). In some implementations, the processor 402 is a single threaded processor. In some implementations, the processor 402 is a multi-threaded processor. In some implementations, the processor 402 is a quantum computer.

[0135] The memory 404 stores information within the computing device 400. In some implementations, the memory 404 is a volatile memory unit or units. In some implementations, the memory 404 is a non-volatile memory unit or units. The memory 404 may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0136] The storage device 406 is capable of providing mass storage for the computing device 400. In some implementations, the storage device 406 may be or include a computer- readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor 402), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine readable mediums (for example, the memory 404, the storage device 406, or memory on the processor 402). The high-speed interface 408 manages bandwidth-intensive operations for the computing device 400, while the low-speed interface 412 manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface 408 is coupled to the memory 404, the display 416 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 410, which may accept various expansion cards (not PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0137] Attorney Docket No. 56113-0734WO1

[0138] Client Ref: GP-307788-00-PCT shown). In the implementation, the low-speed interface 412 is coupled to the storage device 406 and the low-speed expansion port 414. The low-speed expansion port 414, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g. , through a network adapter.

[0139] The computing device 400 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 420, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer 422. It may also be implemented as part of a rack server system 424. Alternatively, components from the computing device 400 may be combined with other components in a mobile device, such as a mobile computing device 450. Each of such devices may include one or more of the computing device 400 and the mobile computing device 450, and an entire system may be made up of multiple computing devices communicating with each other.

[0140] The mobile computing device 450 includes a processor 452, a memory 464, an input / output device such as a display 454, a communication interface 466, and a transceiver 468, among other components. The mobile computing device 450 may also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor 452, the memory 464, the display 454, the communication interface 466, and the transceiver 468, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0141] The processor 452 can execute instructions within the mobile computing device 450, including instructions stored in the memory 464. The processor 452 may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 452 may provide, for example, for coordination of the other components of the mobile computing device 450, such as control of user interfaces, applications run by the mobile computing device 450, and wireless communication by the mobile computing device 450.

[0142] The processor 452 may communicate with a user through a control interface 458 and a display interface 456 coupled to the display 454. The display 454 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 456 may include appropriate circuitry for driving the display 454 to PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0143] Attorney Docket No. 56113-0734WO1

[0144] Client Ref: GP-307788-00-PCT present graphical and other information to a user. The control interface 458 may receive commands from a user and convert them for submission to the processor 452. In addition, an external interface 462 may provide communication with the processor 452, so as to enable near area communication of the mobile computing device 450 with other devices. The external interface 462 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0145] The memory 464 stores information within the mobile computing device 450. The memory 464 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 474 may also be provided and connected to the mobile computing device 450 through an expansion interface 472, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 474 may provide extra storage space for the mobile computing device 450, or may also store applications or other information for the mobile computing device 450. Specifically, the expansion memory 474 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory 474 may be provide as a security module for the mobile computing device 450, and may be programmed with instructions that permit secure use of the mobile computing device 450. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0146] The memory may include, for example, flash memory and / or NVRAM memory (nonvolatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier such that the instructions, when executed by one or more processing devices (for example, processor 452), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory 464, the expansion memory 474, or memory on the processor 452). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver 468 or the external interface 462.

[0147] The mobile computing device 450 may communicate wirelessly through the communication interface 466, which may include digital signal processing circuitry in some cases. The communication interface 466 may provide for communications under PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0148] Attorney Docket No. 56113-0734WO1

[0149] Client Ref: GP-307788-00-PCT various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), LTE, 4G / 6G cellular, among others. Such communication may occur, for example, through the transceiver 468 using a radio frequency. In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module 470 may provide additional navigation- and location-related wireless data to the mobile computing device 450, which may be used as appropriate by applications running on the mobile computing device 450.

[0150] The mobile computing device 450 may also communicate audibly using an audio codec 460, which may receive spoken information from a user and convert it to usable digital information. The audio codec 460 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device 450. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, among others) and may also include sound generated by applications operating on the mobile computing device 450.

[0151] The mobile computing device 450 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 480. It may also be implemented as part of a smart-phone 482, personal digital assistant, or other similar mobile device.

[0152] In general, use of "or" can refer to “and / or.” When providing a list of two or more items, the conjunction “or” can indicate any one of the items, any combination of a subset of the items, or all items in combination.

[0153] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0154] The subject matter and the actions and operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0155] Attorney Docket No. 56113-0734WO1

[0156] Client Ref: GP-307788-00-PCT firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter and the actions and operations described in this specification can be implemented as or in one or more computer programs, e.g., one or more modules of computer program instructions, encoded on a computer program carrier, for execution by, or to control the operation of, data processing apparatus. The earner can be a tangible non-transitory computer storage medium. Alternatively or in addition, the carrier can be an artificially- generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be or be part of a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. A computer storage medium is not a propagated signal.

[0157] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. Data processing apparatus can include special-purpose logic circuitry, e g., an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or a GPU (graphics processing unit). The apparatus can also include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0158] A computer program can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program, e.g., as an app, or as a module, component, engine, subroutine, or other unit suitable for executing in a computing environment, which environment may include one or more computers interconnected by a data communication network in one or more locations.

[0159] A computer program may, but need not, correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0160] Attorney Docket No. 56113-0734WO1

[0161] Client Ref: GP-307788-00-PCT

[0162] The processes and logic flows described in this specification can be performed by one or more computers executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuitry, e.g., an FPGA, an ASIC, or a GPU, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0163] Computers suitable for the execution of a computer program can be based on general or special-purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a readonly memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory' can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0164] Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices, and be configured to receive data from or transfer data to the mass storage devices. The mass storage devices can be, for example, magnetic, magneto-optical, or optical disks, or solid state drives. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0165] To provide for interaction with a user, the subject matter described in this specification can be implemented on one or more computers having, or configured to communicate with, a display device, e.g., a LCD (liquid crystal display) monitor, or a virtual-reality (VR) or augmented-reality (AR) display, for displaying information to the user, and an input device by which the user can provide input to the computer, e.g., a keyboard and a pointing device, e.g., a mouse, a trackball or touchpad. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback and responses provided to the user can be any form of sensory feedback, e.g., visual, auditory, speech, or tactile feedback or responses; and input from the user can be received in any form, including acoustic, speech, tactile, or eye tracking input, including touch motion or gestures, or kinetic motion or gestures or orientation motion or gestures. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser, or PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0166] Attorney Docket No. 56113-0734WO1

[0167] Client Ref: GP-307788-00-PCT by interacting with an app running on a user device, e.g., a smartphone or electronic tablet. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0168] This specification uses the term “configured to” in connection with systems, apparatus, and computer program components. That a system of one or more computers is configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. That one or more computer programs is configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. That special-purpose logic circuitry is configured to perform particular operations or actions means that the circuitry has electronic logic that performs the operations or actions.

[0169] The subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0170] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0171] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what is being claimed, which is PCT / US24 / 41705 09 August 2024 (09.08.2024)

[0172] Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT defined by the claims themselves, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claim may be directed to a subcombination or variation of a subcombination.

[0173] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this by itself should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and sy stems can generally be integrated together in a single software product or packaged into multiple software products.

[0174] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0175] What is claimed is:

Claims

PCT / US24 / 41705 09 August 2024 (09.08.2024)Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCTCLAIMS1. A method comprising: assigning a first virtual machine (VM) to a first image capture device and a second VM to a second image capture device , wherein each of the first and second VMs execute on a computing device communicably coupled to the first and second image capture devices and that includes a device memory and a plurality of image signal processor (ISP) blocks; allocating a first portion of the device memory to the first VM and a second portion of the device memory to the second VM; storing, by the first VM, processing parameters of a first ISP block in the first portion of the device memory; providing, by the first VM, data indicating a location of the first portion of the device memory to a microcontroller coupled to the plurality of ISP blocks; transmitting, by the microcontroller and based on the data indicating the location of the first portion of the device memory, the processing parameters from the first portion of the device memory to ISP registers corresponding to the first ISP block; and processing, by the first ISP block, image data from the first image capture device using the processing parameters.

2. The method of claim 1, wherein providing the data indicating the location of the first portion of the device memory to the microcontroller comprises: writing a command to a first mailbox of the microcontroller assigned to the first VM and inaccessible to the second VM.

3. The method of claim 2, wherein writing the command comprises: writing a command configured to generate an interrupt signal.

4. The method of claim 3, wherein transmitting the processing parameters from the first portion of the device memory occurs in response to generating the interrupt signal.

5. The method of claim 1, comprising:PCT / US24 / 41705 09 August 2024 (09.08.2024)Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT loading, by the microcontroller, the processing parameters from the first portion of the device memory; and writing, by the microcontroller, a command to a first mailbox of the microcontroller assigned to the first VM and inaccessible to the second VM upon completion of transmitting the processing parameters from the first portion of the device memory to the ISP registers corresponding to the first ISP block.

6. The method of claim 1, comprising: assigning, by the microcontroller, a (i) first identifier to the first image capture device and (ii) a second identifier to the second image capture device; performing one or more of the following using a memory management unit and the assigned first and second identifiers: blocking the second VM from accessing the processing parameters stored in the first portion of the device memory allocated to the first VM; blocking the second VM from accessing a mailbox of the microcontroller allocated to the first VM; or blocking the second VM from accessing the image data from the first image capture device stored in the first portion of the device memory allocated to the first VM.

7. The method of claim 1 , wherein transmitting the processing parameters from the first portion of the device memory comprises: providing an identifier, with a request for the processing parameters of the first ISP block, to a memory management unit.

8. The method of claim 1, wherein allocating the first portion of the device memory to the first VM comprises: allocating (i) a raw data portion of the device memory to the first VM to store the image data from the first image capture device and (ii) a processed data portion of the device memory to the first VM to store the image data processed by the plurality one or more ISP blocks.

9. The method of claim 8, comprising:PCT / US24 / 41705 09 August 2024 (09.08.2024)Attorney Docket No. 56113-0734WO1Client Ref: GP-307788-00-PCT storing raw image data from the first image capture device within the allocated raw data portion of the device memory; loading, by the microcontroller, the raw image data from the allocated raw data portion of the device memory; and providing the loaded raw image data as the image data to the first ISP block prior to the image data processing by the first ISP block.

10. The method of claim 1, wherein storing the processing parameters of the first ISP block in the first portion of the device memory comprises: storing data indicating one or more of exposure control, white balance correction, auto-focus, noise reduction, sharpness improvement, color correction, or gamma correction.

11. The method of claim 1, comprising: processing, by a second ISP block, second image data after processing the image data by the first ISP block.

12. The method of claim 11, comprising: processing, by a second ISP block, second image data in parallel with processing the image data by the first ISP block.

13. The method of claim 1, comprising: generating a first identifier corresponding to the first image capture device and a second identifier corresponding to the second image capture device.

14. The method of claim 1, wherein the first and second image capture devices are on the device and communicably coupled to the device.

15. The method of claim 1, wherein the first and second image capture devices are on another device different than the device, and wherein the first and second image capture devices of the other device are communicably coupled to the device.PCT / US24 / 41705 09 August 2024 (09.08.2024)Attorney Docket No. 56113-0734WO1 Client Ref: GP-307788-00-PCT16. One or more computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations of any preceding claim.

17. A system comprising: one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations of any preceding claim.

Images