Assistance engine for hardware virtualization

Abstract

Techniques for processing data are described herein. For example, a method may include receiving input data by a symmetric processing engine of two or more symmetric processing engines coupled to an auxiliary processing engine. The method may also include receiving an indication to process the input data using a module of the secondary processing engine; sending the output data to the auxiliary processing engine; receiving processed data from the auxiliary processing engine; further processing the processed data in one or more portions of a pipeline of modules of the symmetric processing engine; and outputting the further processed data.

Description

Technical Field

[0001] The present disclosure generally relates to generating high dynamic range (HDR) images. For example, aspects of the present disclosure relate to systems and techniques for maintaining symmetry of an image front end of an image processing system while reducing the amount of space used by the image processing system to generate a high dynamic range image. Background Art

[0002] A camera is a device that uses an image sensor to receive light and capture image frames such as still images or video frames. The camera may include one or more processors, such as an image signal processor (ISP), which may process one or more image frames captured by the image sensor. For example, a raw image frame captured by a camera sensor may be processed by an image signal processor (ISP) to generate a final image. The camera may be configured with various image capture and image processing settings to change the appearance of the image. Some camera settings are determined and applied before or during image capture, such as ISO, exposure time (also referred to as exposure duration), aperture size, f / stop, shutter speed, focus, and gain. In addition, some camera settings may be configured for post-processing of the image, such as changing contrast, brightness, saturation, sharpness, levels, curves, or color. Summary of the Invention

[0003] Systems and techniques for processing data, such as for processing graphics data, are described herein. In one illustrative example, an apparatus for processing data is provided. The apparatus includes an auxiliary processing engine (e.g., configured in circuitry) and two or more symmetric processing engines coupled to the auxiliary processing engine, each of the two or more symmetric processing engines including a pipeline of processing modules (e.g., configured in circuitry). The symmetric processing engines among the two or more symmetric processing engines are configured to: receive input data; receive an indication to process the input data using a module of the auxiliary processing engine; send output data to the auxiliary processing engine; receive the processed data from the auxiliary processing engine; further process the processed data in one or more portions of the pipeline of modules of the symmetric processing engine; and output the further processed data.

[0004] In another example, a method for processing data includes: receiving input data by a symmetric processing engine among two or more symmetric processing engines coupled to an auxiliary processing engine; receiving an indication to process the input data using a module of the auxiliary processing engine; sending output data to the auxiliary processing engine; receiving the processed data from the auxiliary processing engine; further processing the processed data in one or more portions of the pipeline of modules of the symmetric processing engine; and outputting the further processed data.

[0005] In another example, a non-transitory computer-readable medium is provided that includes instructions that, when executed by one or more processors, cause the one or more processors to: receive input data through a symmetric processing engine of two or more symmetric processing engines coupled to an auxiliary processing engine; receive an indication to process the input data using a module of the auxiliary processing engine; send output data to the auxiliary processing engine; receive the processed data from the auxiliary processing engine; further process the processed data in one or more parts of a pipeline of a module of the symmetric processing engine; and output the further processed data.

[0006] In another example, an apparatus for processing data includes: means for receiving input data through a symmetric processing engine of two or more symmetric processing engines coupled to an auxiliary processing engine; means for receiving an indication to process the input data using a module of the auxiliary processing engine; means for sending output data to the auxiliary processing engine; means for receiving the processed data from the auxiliary processing engine; means for further processing the processed data in one or more parts of a pipeline of a module of the symmetric processing engine; and means for outputting the further processed data.

[0007] In some aspects, each of the apparatuses described above is a mobile device, a smart or connected device, a camera system, and / or an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), or may be a part of or may include these devices. In some examples, the apparatus may include a mobile device (e.g., a mobile phone or a so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotic device or system, or other devices, or is a part of these devices. In some aspects, the apparatus includes one image sensor (e.g., one camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the apparatus includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the apparatus described above may include one or more sensors. In some cases, the one or more sensors may be used to determine the location of the apparatus, the state of the apparatus (e.g., tracking state, operating state, temperature, humidity level, and / or another state), and / or for other purposes.

[0008] The invention content is not intended to identify the key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood with reference to the appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0009] The foregoing, as well as other features and aspects, will become more apparent when referring to the following specification, claims, and appended drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Exemplary examples of the present application are described in detail below with reference to the following drawings:

[0011] Figure 1 is a block diagram illustrating an exemplary architecture of an image processing system according to some examples of the present disclosure;

[0012] Figure 2 is a block diagram illustrating an exemplary system for data processing according to aspects of the present disclosure;

[0013] Figure 3 is a block diagram illustrating an exemplary system for data processing according to aspects of the present disclosure, including an auxiliary engine for virtualizing a front-end module;

[0014] Figure 4 is a block diagram illustrating an enhanced auxiliary engine according to aspects of the present disclosure;

[0015] Figure 5 is a flowchart illustrating a process for processing data according to aspects of the present disclosure;

[0016] Figure 6 illustrates an exemplary computing device architecture of an exemplary computing device that can implement the various technologies described herein. DETAILED DESCRIPTION

[0017] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which will be obvious to those skilled in the art. In the following description, specific details are set forth for the purpose of explanation to provide a thorough understanding of the aspects of the present application. However, it will be obvious that the aspects can be implemented without these specific details. The drawings and description are not intended to be restrictive.

[0018] The following description only provides exemplary aspects and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of the exemplary aspects will provide those skilled in the art with a description that can be used to implement the exemplary aspects. It should be understood that various changes can be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0019] An electronic device (e.g., a mobile phone, a wearable device (e.g., a smart watch, smart glasses, etc.), a tablet computer, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, etc.), a connected device, a laptop computer, etc.) can implement a camera to capture images or video frames of a scene, a person, an animal, and / or any object. A camera can refer to a device that uses an image sensor to receive light and capture image frames (such as still images or video frames). The terms “image,” “image frame,” and “frame” can be used interchangeably herein. A camera system can include a processor (e.g., an image signal processor (ISP), etc.), which can receive one or more images and process the one or more images. For example, a raw image captured by a camera sensor can be processed by an ISP to generate a final image. The processing performed by the ISP can be executed by filters or processing blocks applied to the captured images, such as noise reduction or noise filtering, edge enhancement, color balance, contrast, intensity adjustment (such as darkening or brightening), hue adjustment, etc. An image processing block or module can include lens / sensor noise correction, a Bayer filter, demosaicing, color conversion, correction or enhancement / suppression of image attributes, a noise reduction filter, a sharpening filter, etc.

[0020] Electronic devices (e.g., mobile phones, wearable devices (e.g., smart watches, smart glasses, etc.), tablet computers, extended reality (XR) devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, etc.), connected devices, laptop computers, etc.) are increasingly equipped with camera hardware to capture image frames, such as still images and / or video frames, for consumption. For example, an electronic device can include a camera to allow the electronic device to capture video or images of a scene, a person, an object, etc. A camera is a device that uses an image sensor to receive light and capture image frames (e.g., still images or video frames). In some examples, a camera can include one or more processors, such as an image signal processor (ISP), which can process one or more image frames captured by the image sensor. For example, a raw image frame captured by a camera sensor can be processed by the image signal processor (ISP) of the camera to generate a final image. In some cases, an electronic device implementing a camera can further process the captured image or video for certain effects (e.g., compression, image enhancement, image restoration, scaling, frame rate conversion, etc.) and / or certain applications (such as computer vision, extended reality (e.g., augmented reality, virtual reality, etc.), object detection, image recognition (e.g., face recognition, object recognition, scene recognition, etc.), feature extraction, authentication, and automation, etc.).

[0021] As indicated above, the raw image data captured by an image sensor may be processed by an image processing system such as an ISP to obtain a final output image. In some cases, the image processing system may use multiple stages to process an image. For example, a preprocessing stage may be used to perform real-time preprocessing of full-resolution image data, and the image data may be used, for example, to adjust the image sensor or provide low-level image correction and / or compensation. The preprocessing stage may be performed, for example, by a front-end engine of an image front-end in the image processing system. In some cases, the preprocessed image data may be stored in a memory and then retrieved from the memory for additional processing. The additional processing may be performed on a portion of the preprocessed image rather than on the entire image at once, and the preprocessed image may be streamed from the memory for additional processing. The additional processing may be performed, for example, by an image processor of the image processing system.

[0022] Devices having multiple cameras with multiple image sensors are increasingly being produced. In some cases, the image data from these multiple image sensors may be processed by a common image processing system. To allow the image processing system to process the image data from different image sensors, the image front-end may include multiple front-end engines, and each of the multiple front-end engines may be capable of processing the image data from any of the image sensors. Allowing any front-end engine to process the image data from any image sensor provides flexibility for the image sensors and may reduce the design, testing, and production costs of the front-end engines.

[0023] Since any front-end engine may process the image data from any image sensor, certain preprocessing operations that may be used by fewer than all of the image sensors may be implemented by all of the front-end engines. In some cases, the image front-end may be the largest portion of the image processing system because the image front-end and the front-end engines may operate on full-resolution images and may include buffers large enough to hold the full-resolution images. In some aspects, the front-end engine may include a pipeline of front-end modules configured to perform preprocessing operations for the front-end engine. Additionally, duplicating front-end modules to perform preprocessing operations that may be used by fewer than all of the image sensors incurs an additional area penalty. In some cases, virtualizing the front-end modules that perform preprocessing operations from the front-end engines to a secondary engine that may be shared by multiple front-end engines may be beneficial. This virtualization of certain front-end modules may help reduce or eliminate the area penalty from redundant front-end modules. A module may also be referred to as a block.

[0024] Various aspects of the present application will be described with reference to the accompanying drawings.

[0025] Figure 1is a block diagram illustrating an example architecture of an image processing system 100. The image processing system 100 includes various components for capturing and processing images, such as an image of scene 110. The image processing system 100 can capture image frames (e.g., still images or video frames). In some cases, the lens 115 and the image sensor 130 can be associated with an optical axis. In one illustrative example, both the photosensitive area of the image sensor 130 (e.g., a photodiode) and the lens 115 can be centered on the optical axis.

[0026] In some examples, the lens 115 of the image processing system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the incoming light from the scene towards the image sensor 130. The light received by the lens 115 then passes through the aperture of the image processing system 100. In some cases, the aperture (e.g., aperture size) is controlled by one or more control mechanisms 120. In other cases, the aperture can have a fixed size.

[0027] One or more control mechanisms 120 can control exposure, focus, and / or zoom based on information from the image sensor 130 and / or information from the image processor 150. In some cases, one or more control mechanisms 120 can include multiple mechanisms and components. For example, the control mechanism 120 can include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 can also include additional control mechanisms other than those Figure 1 illustrated. For example, in some cases, one or more control mechanisms 120 can include control mechanisms for controlling analog gain, flash, HDR, depth of field, and / or other image capture characteristics.

[0028] The focus control mechanism 125B of the control mechanism 120 can obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the positioning of the lens 115 relative to the positioning of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to or farther away from the image sensor 130 by actuating a motor or a servo system (or other lens mechanisms) to adjust the focus. In some cases, additional lenses can be included in the image processing system 100. For example, the image processing system 100 can include one or more microlenses on each photodiode of the image sensor 130. These microlenses can each bend the light received from the lens 115 towards the corresponding photodiode before the light reaches the photodiode.

[0029] In some examples, the focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The control mechanism 120, image sensor 130, and / or image processor 150 may be used to determine the focus setting. The focus setting may be referred to as an image capture setting and / or an image processing setting. In some cases, the lens 115 may be fixed relative to the image sensor and the focus control mechanism 125B.

[0030] The exposure control mechanism 125A of the control mechanism 120 may obtain an exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f / stop), the duration the aperture is open (e.g., exposure time or shutter speed), the duration the sensor collects light (e.g., exposure time or electronic shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0031] The zoom control mechanism 125C of the control mechanism 120 can obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servo systems (or other lens mechanisms) to move one or more lenses relative to each other. The zoom setting can be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly can include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly can include a focusing lens (in some cases, the focusing lens can be the lens 115), which first receives light from the scene 110, and the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before reaching the image sensor 130. In some cases, the afocal zoom system can include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference of each other), with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses. In some cases, the zoom control mechanism 125C can control the zoom by capturing an image from an image sensor (e.g., including the image sensor 130) among a plurality of image sensors with a zoom corresponding to the zoom setting. For example, the image processing system 100 can include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a greater zoom. In some cases, based on the selected zoom setting, the zoom control mechanism 125C can capture an image from the corresponding sensor.

[0032] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by the image sensor 130. In some cases, different photodiodes can be covered by different filters. In some cases, different photodiodes can be covered in different color filters and can thus measure light that matches the color of the filter covering the photodiode. Various color filter arrays can be used, such as, for example and without limitation, a Bayer color filter array, a four-color filter array (QCFA), and / or any other color filter array.

[0033] In some cases, the image sensor 130 may alternatively or additionally include opaque and / or reflective masks that block light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, the opaque and / or reflective masks may be used for phase detection autofocus (PDAF). In some cases, the opaque and / or reflective masks may be used to block portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., IR cut-off filter, UV cut-off filter, band-pass filter, low-pass filter, high-pass filter, etc.). The image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output by the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output by the photodiodes (and / or the analog signal amplified by the analog gain amplifier) into a digital signal. In some cases, certain components or functions discussed with respect to one or more of the control mechanisms 120 may alternatively or additionally be included in the image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), complementary metal-oxide semiconductor (CMOS), N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0034] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other type of processor discussed with respect to the computing device architecture 900 of FIG. 9. The host processor 152 may be a digital signal processor (DSP) and / or other type of processor. In some embodiments, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., BluetoothTM, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an inter-integrated circuit 2 (I2C) interface, an inter-integrated circuit 3 (I3C) interface, a serial peripheral interface (SPI) interface, a serial general-purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0035] The image processor 150 may perform multiple tasks, such as demosaicking, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving inputs, managing outputs, managing memory, or some combination thereof. The image processor 150 may store the image frames and / or the processed images in a random access memory (RAM) 140, a read-only memory (ROM) 145, a cache, a memory cell, another storage device, or some combination thereof.

[0036] A variety of input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device, any other input device, or any combination thereof. In some cases, captions may be input into the image processing device 105B through the physical keyboard or keypad of the I / O device 160, or through the virtual keyboard or keypad of the touch screen of the I / O device 160. The I / O devices 160 may include one or more ports, jacks, or other connectors that implement a wired connection between the image processing system 100 and one or more peripheral devices, through which the image processing system 100 may receive data from and / or send data to one or more peripheral devices. The I / O devices 160 may include one or more wireless transceivers that implement a wireless connection between the image processing system 100 and one or more peripheral devices, through which the image processing system 100 may receive data from and / or send data to one or more peripheral devices. The peripheral devices may include any type of I / O device 160 discussed previously, and once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector, they themselves may be considered I / O devices 160.

[0037] In some cases, the image processing system 100 may be a single device. In some cases, the image processing system 100 may be two or more independent devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some specific embodiments, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some specific embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0038] As Figure 1 shown, the vertical dashed line will Figure 1The image processing system 100 is divided into two parts, representing an image capture device 105A and an image processing device 105B respectively. The image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), a RAM 140, a ROM 145, and an I / O device 160. In some cases, certain components exemplified in the image capture device 105A (such as the ISP 154 and / or the host processor 152) may be included in the image capture device 105A. In some examples, the image processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof.

[0039] The image processing system 100 can be a single computing device or part of or implemented by a single computing device or multiple computing devices. In some examples, the image processing system 100 can be part of an electronic device (or multiple electronic devices), such as a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smart phone, cellular phone, conferencing system, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a smart television, a display device, a gaming console, an XR device (e.g., HMD, smart glasses, etc.), an IoT (Internet of Things) device, a smart wearable device, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device.

[0040] The image capture device 105A and the image processing device 105B can be part of the same electronic device or different electronic devices. In some embodiments, the image capture device 105A and the image processing device 105B can be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, a smart phone, a smart television, a gaming console, or other computing devices.

[0041] Although the image processing system 100 is shown as including certain components, those of ordinary skill in the art will understand that the image processing system 100 may include more than Figure 1Those components shown in the figure, and more components. The components of the image processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some specific embodiments, the components of the image processing system 100 may include electronic circuits or other electronic hardware and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of the electronic device implementing the image processing system 100.

[0042] In some examples, Figure 6 the computing device architecture 600 shown and further described below may include the image processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0043] Figure 2 is a block diagram illustrating an example system 200 for data processing according to aspects of the present disclosure. The system 200 may use images captured from any one of the image sensors 202A, 202B, ……, 202J (collectively referred to as the image sensors 202) to generate a final image 220. In some cases, the image sensor 202 may be Figure 1 the image sensor 130. In some cases, the final image 220 may be generated from images captured by multiple image sensors 202. The image sensors 202 may be coupled to T front-end engines 204A, 204B, ……, 204T (collectively referred to as the front-end engines 206) of the image front-end 204 via a sensor crossbar matrix 208. The image crossbar matrix 208 may couple each of the image sensors 202 to each of the front-end engines 204 of the image front-end 206. In some cases, since each of the image sensors 202 is coupled to each of the front-end engines 204 of the image front-end 206, any one of the front-end engines 204 may be used to preprocess the image data from any one of the image sensors 202. In some examples, the number of front-end engines 204 included in the image front-end 206 may be based on the number of image sensors 202 that can be used concurrently.

[0044] The image front end 206 can perform real-time preprocessing on full-resolution image data such as that captured by the image sensor 202. To allow preprocessing of full-resolution images, the image front end 206 can include a large image buffer (not shown) sufficient to store one or more full-resolution images such as those captured by the image sensor 202. In some cases, the image front end engine 204 of the image front end 206 can perform one or more preprocessing operations. The preprocessing operations can include, for example but not limited to, autofocus operations, chromatic aberration correction, pixel luminance transformation (e.g., brightness correction, grayscale transformation, etc.), color space conversion, geometric transformation (e.g., rotation, scaling, translation, affine transformation, resizing, etc.), image filtering (e.g., image and / or edge smoothing and / or enhancement, noise reduction, image sharpening, etc.), image warping, image segmentation, image restoration, image enhancement, lens shading, color correction, black level adjustment, lens distortion correction, defective pixel replacement, demosaicing, color balance, compression, interpolation, any other image preprocessing operations, and / or combinations thereof. In some cases, the operations that can be performed by the front end engine 204 as preprocessing operations can be those that should be performed with relatively low latency such as those for phase detection autofocus, or those that can produce higher image quality when performed at the beginning of the image processing pipeline such as those for chromatic aberration correction.

[0045] In some cases, the front end engine 204 can include a processing pipeline of N (e.g., two or more) front end modules 210A, 210B, ……, 210N (collectively referred to as front end modules 210), which can apply one or more preprocessing operations. For a first example, a front end module such as front end module 1 210A can preprocess an image from the image sensor 202 to perform a phase detection autofocus operation. As a second example, another front end module such as front end module 2 210B can perform a chromatic aberration correction operation. In some cases, since any of the front end engines 204 can be used to preprocess the image data from any of the image sensors 202, the front end engines 204 can all be the same and include the same front end modules 210. Having the same image front end engine 204 can help reduce design, testing, and production costs because there is no need to develop, test, and / or produce individual front end engines for different types of image sensors. Additionally, the same image front end engine 204 can allow designers to more flexibly select the image sensor 202 and customize features for different end products without having to consider different types of front end engines 204.

[0046] In some cases, although the front-end engine 204 may fully include front-end modules 210 for preprocessing images from any of the image sensors 202, images from some of the image sensors 202 may not be preprocessed by some of the front-end modules 210 of the front-end engine 204. Referring to the first example above, an image sensor such as image sensor 202A may support phase detection, and thus an image generated by the image sensor 202A may be preprocessed by a front-end module such as front-end module 1 210A that performs a phase-detection autofocus operation. Another image sensor such as image sensor 202B may not support phase detection, and thus an image generated by the image sensor 202B may not be preprocessed by a front-end module such as front-end module 1 210A that performs a phase-detection autofocus operation. Referring to the second example above, a lateral chromatic aberration correction operation may be performed, for example, by front-end module 2210B on an image generated by an ultra-wide-angle sensor such as image sensor 202B, but not on an image generated by other image sensors such as image sensor 202A. In some examples, the front-end modules 210 may be software-controlled such that an appropriate front-end module 210 is selected based on which image sensor 202 is providing image data.

[0047] After the front-end module 210 preprocesses an image from the image sensor 202, the preprocessed image may be written to the memory 212. The image processor 214 may perform offline (e.g., memory-to-memory) processing on the preprocessed image data from the memory 212. Since the image processor 214 does not need to process the image in real time, the image processor 214 may process the image in parts, e.g., by reading a part (e.g., a strip of image data) of the preprocessed image, processing that part, writing the processed part back to the memory 212, and then processing the next part of the preprocessed image. After processing all parts of the image, the processed parts of the final image 220 may be retrieved and output from the memory 212. Since the image processor 214 may be able to process the image in parts rather than processing the entire image at once, the image processor 214 may include a relatively smaller amount of buffer memory as compared to the image front end 206. Additionally, since the image processor 214 may not process the image in real time, fewer image processors 214 may be used as compared to several image sensors that may operate concurrently. In some cases, a single image processor 214 may be used with multiple image sensors 202.

[0048] In some cases, the image processor 214 may include a processing pipeline that includes any number of image processor modules 216A, …, 216P for performing one or more processing operations on the pre-processed image data. One or more processing operations may include, for example but not limited to, filtering operations, blending operations (e.g., blending pixel values), and / or interpolation operations, pixel luminance transformation (e.g., brightness correction, grayscale transformation, etc.), color space conversion, geometric transformation (e.g., rotation, scaling, translation, affine transformation, resizing, etc.), cropping operations, white balance operations, noise reduction operations, image sharpening operations, chroma sampling, image scaling, lens correction operations, segmentation operations, filtering operations (e.g., filtering for adjusting image quality in terms of contrast, noise, texture, resolution, etc.), image warping operations, image restoration operations, lens shading operations, lens distortion correction operations, defective pixel replacement operations, demosaicing operations, color balance operations, smoothing operations, image enhancement operations, operations for implementing image effects or style adjustments, feature enhancement operations, image scaling or resizing operations, color correction operations, black level adjustment operations, linearization operations, gamma correction operations, any other image post-processing operations, and / or combinations thereof.

[0049] In some examples, one or more components of the system 200 may be integrated into a single chip or integrated circuit. For example, the sensor crossbar matrix 208, the image front end 206, the memory 212, and the image processor 214 may be integrated into a single image signal processor. As another example, the components of the system 200 may be integrated into a SoC, as discussed above with respect to Figure 1 discussed.

[0050] As indicated above, since any of the image front end engines 204 may pre-process images from any image sensor 202, the image front end engines 204 may be the same (e.g., including the same front end modules). One issue with having the same image front end engines is that certain operations may be used by fewer than all of the image sensors 202. Returning to the previous example, the phase detection autofocus operation and the chromatic aberration correction operation may be used by fewer than all of the image sensors 202 (e.g., only one or even none of the image sensors 202). In such cases, redundant front end modules that repeatedly perform such operations across all of the image front end engines 204 incur an area penalty when the physical size of the front end engine 204 is increased to include circuitry for the corresponding redundant front end modules. In some cases, the redundant front end modules may be virtualized across the front end engines to help reduce or eliminate the area penalty.

[0051] Figure 3 is a block diagram illustrating an example system 300 for data processing that includes an auxiliary engine for virtualizing front end modules in accordance with aspects of the present disclosure. In system 300, similar toFigure 2 In system 200, two or more image sensors 302A, 302B, ……, 302J (collectively image sensors 302) can transfer images to image front end 306 via sensor crossbar matrix 308 for preprocessing by a set of T front end engines 304A, 304B, ……, 304T (collectively front end engines 304). Similar to system 200, in system 300, image front end 306 can perform real-time preprocessing on full-resolution image data captured by image sensors 302. Front end engines 304 include a processing pipeline of two or more front end modules 310A, ……, 310N for performing preprocessing operations.

[0052] Image front end 306 also includes an auxiliary engine 322 coupled to front end engines 304. In some cases, in addition to front end modules 310, front end engines 304 may also include a multiplexer 326 and a demultiplexer 328. Front end engines 304 can be coupled to a multiplexer 330 of auxiliary engine 322 via multiplexer 326 and demultiplexer 328.

[0053] In some cases, auxiliary engine 322 may include a multiplexer 330 and any number K of auxiliary engine modules 324A, ……, 324K (collectively auxiliary engine modules 324). Similar to front end modules 310, auxiliary engine modules 324 can perform preprocessing operations on full-resolution image data captured by image sensors 302. In some cases, auxiliary engine modules 324 can be implemented in a manner substantially similar to front end modules 310, and front end modules 310 that perform preprocessing operations used by less than all image sensors 302 can be implemented (e.g., moved) in auxiliary engine 322. Referring to the example discussed above in conjunction with Figure 2 the example, the lateral chromatic aberration correction operation can be used by a single image sensor such as an ultra-wide-angle image sensor, and the lateral chromatic aberration correction operation can be performed, for example, by front end module 2 210B. This front end module may not be implemented in front end engines 304 of system 300. Instead, the preprocessing operations performed by front end module 2 210B can be implemented in auxiliary engine 322 as auxiliary engine module 324, such as auxiliary engine module 1 324A.

[0054] In some cases, the dig-out and dig-in points from the front-end engine 304 to the auxiliary engine 322 may replace front-end modules implemented in the auxiliary engine 322 (e.g., moved to the auxiliary engine), such as front-end module 2 210B. In some cases, when a series of image preprocessing operations are performed in a specific order, the image quality may be enhanced. This order may define how the front-end modules 310 are sorted in the front-end engine 304. As an example, when the lateral chromatic aberration correction operation is performed by front-end module 2 210B after other preprocessing operations are performed by front-end module 1 210A, the image quality may be enhanced. Other operations may also be performed after the lateral chromatic aberration correction operation. To help preserve this order of operations, if the lateral chromatic aberration correction operation is implemented in an auxiliary engine module such as auxiliary engine module 1 324A, the dig-out and dig-in from the front-end engine 304 to the auxiliary engine 322 may be placed where the front-end module implementing the lateral chromatic aberration correction operation would be placed. In some cases, the dig-out and dig-in from the front-end engine 304 to the auxiliary engine 322 may exist in all front-end engines 304 of the image front-end 306. In some cases, the dig-out functionality may be provided by the multiplexer 326, and the dig-in functionality may be provided by the demultiplexer 328.

[0055] In some cases, the multiplexer 326 and the demultiplexer 328 may be software-controlled to direct the data stream to the auxiliary engine 322 when an auxiliary engine module 324 is to be used. For example, the data from front-end module 1 310A will be output via the multiplexer 326 to the appropriate auxiliary engine module 324 through the multiplexer 330 of the auxiliary engine 322.

[0056] In some cases, the multiplexer 330 of the auxiliary engine 322 may select the input from the appropriate front-end engine 304. In some cases, the multiplexer 330 and the auxiliary engine module 324 may also be software-controlled to select the appropriate input from among the front-end engines 304 and to select the appropriate auxiliary engine module 324. In some cases, the image data from one front-end engine 304 may be input to the auxiliary engine 322 at a time. In other cases, the image data from multiple front-end engines 304 may be input to the auxiliary engine 322 at a time. How many front-end engines 304 the auxiliary engine 322 can receive input from at a time may depend on which auxiliary engine modules 324 are to be used and / or the availability of the auxiliary engine modules 324. In some cases, the auxiliary engine 322 may have multiple multiplexers 330. For example, the auxiliary engine 322 may have multiplexers 330 and auxiliary engine modules 324 combined for each preprocessing operation supported by the auxiliary engine 322.

[0057] After being pre - processed by a suitable auxiliary engine module 324, the auxiliary engine 322 can output the image data to the next suitable front - end module 310 (if any) through the demultiplexer 328 of the front - end engine 1304A. If the auxiliary engine 322 does not perform additional pre - processing operations, the front - end engine 304A can continue to pre - process the image data in a manner similar to the front - end engine 204A.

[0058] In some cases, the clock rate of the auxiliary engine 322 can be the same as or faster than the maximum clock rate of the front - end engine 304. Thus, the auxiliary engine 322 can better maintain the performance of the front - end engine 304 and mitigate potential cross - clock - domain issues.

[0059] In some cases, the auxiliary engine 322 can be bypassed. For example, image data from a non - ultra - wide - angle image sensor may not require performing lateral chromatic aberration correction operations. In this example, the image data can be input from the front - end module 1 310A to the multiplexer 326 and directly passed to the demultiplexer 328 and then continue to be passed to the next suitable front - end module 310 (if any).

[0060] In some cases, parameterized pipeline stages can be placed between one or more front - end engines 304 and the auxiliary engine. These parameterized pipeline stages can be adjusted for each front - end engine 304 based on how far the particular front - end engine 304 is physically from the auxiliary engine 322.

[0061] Figure 4 is a block diagram illustrating an enhanced auxiliary engine 400 according to aspects of the present disclosure. In some cases, pre - processing operations that are used by less than all image sensors concurrently can also be implemented in the enhanced auxiliary engine 400. As an example, a device can include multiple image sensors, all of which use phase - detection autofocus operations, but the device is configured (e.g., based on availability issues and / or feature differentiation / market concerns) such that less than all (e.g., two) image sensors can be used concurrently. Implementing the phase - detection autofocus operation in the front - end engine while maintaining the ability of any image sensor to use any front - end engine can result in a phase - detection autofocus front - end engine module in each front - end engine. Instead of implementing the phase - detection autofocus operation in each front - end engine, it can be implemented in the enhanced auxiliary engine based on the expected concurrent use cases (e.g., using two image sensors concurrently). For example, the enhanced auxiliary engine 400 can be enhanced to dynamically and concurrently direct multiple data streams to one or more auxiliary engine modules 402A, 402B, ……, 402O (collectively referred to as auxiliary engine modules 424).

[0062] In the enhanced assist engine 400, a first input 404A from one of the front-end engines (not shown) can be directed by the assist engine input crossbar matrix 406 to a first assist engine module, such as assist engine module 2 402B. The output of assist engine module 2 402B can be passed to the assist output crossbar matrix 408, which directs the output back to the appropriate front-end engine. Concurrently, a second input 404B from another front-end engine in the front-end engines can be directed by the assist engine input crossbar matrix 406 to a second assist engine module, such as assist engine module 1 402A. The output of assist engine module 1 402A can also be passed to the assist output crossbar matrix 408, which directs the output back to the appropriate front-end engine. In the example discussed above, since the expected concurrent use case is for two concurrent phase detection autofocus operations, the third assist engine module 3 402C shown in the enhanced assist engine 400 can be omitted.

[0063] In some cases, the enhanced assist engine 400 can also be used to implement variations of the preprocessing operations. As an example, different assist engine modules 402 can perform the phase detection autofocus operation in different ways. The enhanced assist engine 400 can include as many different assist engine modules 402 for performing the phase detection autofocus operation as needed. In this example, an assist engine module such as assist engine module 1 402A can perform a full-featured phase detection autofocus operation, and another assist engine module such as assist engine module 2 402B can perform a lightweight version of the phase detection autofocus operation. In some cases, different versions of the enhanced assist engine 400 can include different assist engine modules based on, for example, the image sensor and / or feature differentiation / market focus. In some cases, data from any image sensor can be directed to any appropriate assist engine module 402 for preprocessing.

[0064] In some cases, the techniques discussed above with respect to the enhanced assist engine 400 can be used in place of or in combination with the techniques discussed with respect to Figure 3 the assist engine 322. In some cases, the techniques discussed with respect to the assist engine 322 and the enhanced assist engine 40 can be applied to other processing workflows having parallel symmetric processing engines, such as for image processing and / or graphics processing.

[0065] Figure 5FIG. 500 is a flow chart illustrating a process 500 for processing data in accordance with aspects of the present disclosure. Process 500 may be performed by a computing device (or apparatus) or a component of a computing device (e.g., a chipset, a codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, or other types of computing devices. The operations of process 500 may be implemented as software components executed and run on one or more processors.

[0066] At block 502, the computing device (or its component) may receive input data via a symmetric processing engine among two or more symmetric processing engines coupled to an auxiliary processing engine. In some cases, the input data is received from two or more image sensors for processing by any one of the two or more symmetric processing engines. In some cases, the computing device (or its component) may receive output data from one of the two or more symmetric processing engines at a time. In some cases, the computing device (or its component) may switch between receiving output data from a symmetric processing engine and receiving output data from an additional symmetric processing engine among the two or more symmetric processing engines.

[0067] At block 504, the computing device (or its component) may receive an indication to process the input data using a module of the auxiliary processing engine. In some cases, the computing device (or its component) may perform one or more data processing operations used by less than all of the two or more image sensors. In some cases, the computing device (or its component) may receive output data from the two or more symmetric processing engines concurrently. In some cases, the auxiliary processing engine includes a certain number of modules for performing data processing operations, where the number of modules is based on the number of the two or more symmetric processing engines from which the auxiliary processing engine may receive output data concurrently. In some cases, the auxiliary processing engine includes one or more crossbar switch matrices, and each crossbar switch matrix in the one or more crossbar switch matrices is configured to couple one or more image sensors to each of the two or more symmetric processing engines.

[0068] At block 506, the computing device (or its component) may send the output data to the auxiliary processing engine. At block 508, the computing device (or its component) may receive the processed data from the auxiliary processing engine. At block 510, the computing device (or its component) may further process the processed data in one or more parts of a pipeline of a module of the symmetric processing engine to generate further processed data. At block 512, the computing device (or its component) may output the further processed data.

[0069] Figure 6Example computing device architecture 600 that illustrates an example computing device that can implement the various technologies described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other devices. For example, the computing device architecture 600 may include Figure 1 an image processing system 100. The components of the computing device architecture 600 are shown to communicate with each other electrically using a connector 605 such as a bus. The example computing device architecture 600 includes a processing unit (CPU or processor) 610 and a computing device connector 605 that couples various computing device components including a computing device memory 615 such as a read-only memory (ROM) 620 and a random access memory (RAM) 625 to the processor 610.

[0070] The computing device architecture 600 may include a cache of high-speed memory that is directly connected to, very close to, or integrated as part of the processor 610. The computing device architecture 600 may copy data from the memory 615 and / or the storage device 630 to the cache 612 for quick access by the processor 610. In this way, the cache can provide a performance boost that avoids delays while the processor 610 waits for data. These modules and other modules may control or be configured to control the processor 610 to perform various actions. Other computing device memories 615 may also be used. The memory 615 may include a variety of different types of memory with different performance characteristics. The processor 610 may include any general-purpose processor and hardware or software services configured to control the processor 610 such as service 1 632, service 2 634, and service 3 636 stored in the storage device 630, and a dedicated processor in which software instructions are incorporated into the processor design. The processor 610 may be a self-contained system that includes multiple cores or processors, buses, memory controllers, caches, etc. A multi-core processor may be symmetric or asymmetric.

[0071] To enable user interaction with the computing device architecture 600, the input device 645 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The output device 635 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multimodal computing device enables a user to provide multiple types of input to communicate with the computing device architecture 600. The communication interface 640 generally governs and manages user input and computing device output. There are no limitations on operating on any particular hardware arrangement, and thus the underlying features here can be easily replaced with improved hardware or firmware arrangements as they are developed.

[0072] The storage device 630 is non-volatile memory and can be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic tape cassette, a flash memory card, a solid-state memory device, a digital versatile disc, a cassette tape, a random access memory (RAM) 625, a read-only memory (ROM) 620, and their hybrid forms. The storage device 630 can include services 632, 634, 636 for controlling the processor 610. Other hardware or software modules are contemplated. The storage device 630 can be connected to the computing device via the connector 605. In one aspect, a hardware module that performs a specific function can include software components stored in a computer-readable medium connected to necessary hardware components such as the processor 610, the connector 605, the output device 635, etc., to perform the function.

[0073] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, a smart phone, a tablet computer, a laptop computer, a vehicle, a drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although the following is described with respect to a device having or coupled to one light projector, aspects of the present disclosure can be applied to devices having any number of light projectors and are thus not limited to a particular device.

[0074] The term "device" is not limited to one or a specific number of physical objects (such as a smart phone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or a specific implementation. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0075] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, one of ordinary skill in the art will understand that the embodiments can be practiced without these specific details. For clarity, in some cases, the technology may be presented as including separate functional blocks, including functional blocks that include devices, device components, steps or routines in a method embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein can be used. For example, circuits, systems, networks, processes, and other components can be shown as components in block diagram form to avoid obscuring these embodiments in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details to avoid confusing the embodiments.

[0076] Individual embodiments may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many of the operations can be performed in parallel or concurrently. Additionally, the order of the operations can be rearranged. A process is terminated when the operations of the process are completed, but a process can have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process can correspond to the function returning to the calling function or the main function.

[0077] The processes and methods according to the examples described above can be implemented using computer-executable instructions stored or otherwise obtained from a computer-readable medium. Such instructions can include, for example, instructions and data that configure or otherwise cause a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used can be accessed through a network. The computer-executable instructions can be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0078] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. A computer-readable medium may include non-transitory media in which data can be stored and that do not include carrier waves and / or transient electronic signals propagated wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media (such as flash memory), memories or memory devices, magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, compact discs (CDs) or digital versatile discs (DVDs), any suitable combination thereof, and the like. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. The information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, and the like.

[0079] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals that include bitstreams and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0080] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any form factor among a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may execute the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, and the like. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By additional example, such functionality may also be implemented on a circuit board among different chips or different processes executed on a single device.

[0081] Instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0082] In the foregoing description, aspects of the present application have been described with reference to specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and employed in various other ways, and the appended claims are intended to be construed to include such variations, unless limited by the prior art. The various features and aspects of the applications described above can be used singly or in combination. Additionally, the embodiments can be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. For purposes of illustration, the methods are described in a particular order. It is to be understood that in alternative embodiments, the methods can be performed in a different order than that described.

[0083] One of ordinary skill in the art will appreciate that the less than (“<”) and greater than (“>”) symbols or terms used herein can be replaced with less than or equal to (“≤”) and greater than or equal to (“≥”) symbols, respectively, without departing from the scope of the specification.

[0084] In cases where a component is described as “configured to” perform certain operations, such configuration can be implemented, for example, by designing electronic circuitry or other hardware to perform the operations, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operations, or any combination thereof.

[0085] The phrase “coupled to” means that any component is directly or indirectly physically connected to another component, and / or any component directly or indirectly communicates with another component (e.g., is connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0086] Claim language or other language reciting “at least one of” a set and / or “one or more of” a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language “at least one of” a set and / or “one or more of” a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.

[0087] The various illustrative logical blocks, modules, circuits, and algorithmic steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or any combination thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0088] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be at least partially implemented by a computer-readable data storage medium comprising program code, including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. Additionally or alternatively, the techniques may be at least partially implemented by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0089] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. The general-purpose processor may be a microprocessor; but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0090] Exemplary aspects of the present disclosure include:

[0091] Aspect 1. An apparatus for processing data, the apparatus comprising: an auxiliary processing engine; and two or more symmetric processing engines coupled to the auxiliary processing engine, each of the two or more symmetric processing engines including a pipeline of processing modules, wherein a symmetric processing engine of the two or more symmetric processing engines is configured to: receive input data; receive an indication to process the input data using a module of the auxiliary processing engine; send output data to the auxiliary processing engine; receive processed data from the auxiliary processing engine; further process the processed data in one or more portions of the pipeline of the symmetric processing engine's module; and output the further processed data.

[0092] Aspect 2. The apparatus according to aspect 1, wherein the input data is received from two or more image sensors for processing by any one of the two or more symmetric processing engines.

[0093] Aspect 3. The apparatus according to aspect 2, wherein the auxiliary processing engine is configured to perform one or more data processing operations used by less than all of the two or more image sensors.

[0094] Aspect 4. The apparatus according to any one of aspects 1 to 3, wherein the auxiliary processing engine is configured to receive output data from one of the two or more symmetric processing engines at a time.

[0095] Aspect 5. The apparatus according to aspect 4, wherein the auxiliary processing engine is configured to switch between receiving output data from the symmetric processing engine and receiving output data from an additional symmetric processing engine among the two or more symmetric processing engines.

[0096] Aspect 6. The apparatus according to any one of aspects 1 to 5, wherein the auxiliary processing engine is configured to receive output data from the two or more symmetric processing engines concurrently.

[0097] Aspect 7. The apparatus according to aspect 6, wherein the auxiliary processing engine includes a certain number of modules for performing data processing operations, and the number of the modules is based on the number of the two or more symmetric processing engines from which the auxiliary processing engine can receive output data concurrently.

[0098] Aspect 8. The apparatus according to aspect 6, wherein the auxiliary processing engine includes one or more crossbar switch matrices, and each crossbar switch matrix in the one or more crossbar switch matrices is configured to couple one or more image sensors to each of the two or more symmetric processing engines.

[0099] Aspect 9. A method for processing data, the method comprising: receiving input data by a symmetric processing engine among two or more symmetric processing engines coupled to an auxiliary processing engine; receiving an indication to process the input data using modules of the auxiliary processing engine; sending output data to the auxiliary processing engine; receiving the processed data from the auxiliary processing engine; further processing the processed data in one or more parts of a pipeline of modules of the symmetric processing engine; and outputting the further processed data.

[0100] Aspect 10. The method according to aspect 9, wherein the input data is received from two or more image sensors for processing by any one of the two or more symmetric processing engines.

[0101] Aspect 11. The method according to aspect 10, the method further comprising: performing one or more data processing operations used by less than all of the two or more image sensors.

[0102] Aspect 12. The method according to any one of aspects 9 to 11, the method further comprising: receiving output data from one symmetric processing engine among the two or more symmetric processing engines at a time.

[0103] Aspect 13. The method according to aspect 12, the method further comprising: switching between receiving output data from the symmetric processing engine and receiving output data from an additional symmetric processing engine among the two or more symmetric processing engines.

[0104] Aspect 14. The method according to any one of aspects 9 to 13, the method further comprising: concurrently receiving output data from the two or more symmetric processing engines.

[0105] Aspect 15. The method according to aspect 14, wherein the auxiliary processing engine includes a certain number of modules for performing data processing operations, wherein the number of modules is based on the number of the two or more symmetric processing engines from which the auxiliary processing engine can concurrently receive output data.

[0106] Aspect 16. The method according to aspect 14, wherein the auxiliary processing engine includes one or more crossbar switch matrices, and each crossbar switch matrix in the one or more crossbar switch matrices is configured to couple one or more image sensors to each of the two or more symmetric processing engines.

[0107] Aspect 17. An apparatus for processing data, the apparatus comprising: means for receiving input data through a symmetric processing engine in two or more symmetric processing engines coupled to an auxiliary processing engine; means for receiving an indication to process the input data using modules of the auxiliary processing engine; means for sending output data to the auxiliary processing engine; means for receiving processed data from the auxiliary processing engine; means for further processing the processed data in one or more parts of a pipeline of modules of the symmetric processing engine; and means for outputting the further processed data.

[0108] Aspect 18. The apparatus according to aspect 17, wherein the input data is received from two or more image sensors for processing by any one of the two or more symmetric processing engines.

[0109] Aspect 19. The apparatus according to aspect 18, the apparatus further comprising: means for performing one or more data processing operations used by less than all of the two or more image sensors.

[0110] Aspect 20. The apparatus according to any one of aspects 17 to 19, the apparatus further comprising means for receiving output data from one of the two or more symmetric processing engines at a time.

[0111] Aspect 21. The apparatus according to aspect 20, the apparatus further comprising means for switching between receiving output data from the symmetric processing engine and receiving output data from an additional symmetric processing engine among the two or more symmetric processing engines.

[0112] Aspect 22. The apparatus according to any one of aspects 17 to 21, the apparatus further comprising: means for concurrently receiving output data from the two or more symmetric processing engines.

[0113] Aspect 23. The apparatus according to aspect 22, wherein the auxiliary processing engine includes a certain number of modules for performing data processing operations, wherein the number of the modules is based on the number of the two or more symmetric processing engines from which the auxiliary processing engine can concurrently receive output data.

[0114] Aspect 24. The method according to aspect 22, wherein the auxiliary processing engine includes one or more crossbar switch matrices, and each of the one or more crossbar switch matrices is configured to couple one or more image sensors to each of the two or more symmetric processing engines.

[0115] Aspect 25. A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of aspects 1 to 24.

Claims

1. An apparatus for processing data, the apparatus comprising: An auxiliary processing engine; And Two or more symmetric processing engines, the two or more symmetric processing engines being coupled to the auxiliary processing engine, each of the two or more symmetric processing engines including a pipeline of processing modules, wherein a symmetric processing engine among the two or more symmetric processing engines is configured to: Receive input data; Receive an indication to process the input data using a module of the auxiliary processing engine; Send output data to the auxiliary processing engine; Receive processed data from the auxiliary processing engine; Further process the processed data in one or more parts of the pipeline of the modules of the symmetric processing engine to generate further processed data; And Output the further processed data.

2. The apparatus according to claim 1, wherein the input data is received from two or more image sensors for processing by any one of the two or more symmetric processing engines.

3. The apparatus according to claim 2, wherein the auxiliary processing engine is configured to perform one or more data processing operations used by less than all of the two or more image sensors.

4. The apparatus according to claim 1, wherein the auxiliary processing engine is configured to receive output data from one of the two or more symmetric processing engines at a time.

5. The apparatus according to claim 4, wherein the auxiliary processing engine is configured to switch between receiving output data from the symmetric processing engine and receiving output data from an additional symmetric processing engine among the two or more symmetric processing engines.

6. The apparatus according to claim 1, wherein the auxiliary processing engine is configured to receive output data from the two or more symmetric processing engines concurrently.

7. The apparatus according to claim 6, wherein the auxiliary processing engine includes a certain number of modules for performing data processing operations, wherein the number of modules is based on the number of the two or more symmetric processing engines from which the auxiliary processing engine can concurrently receive output data.

8. The apparatus according to claim 6, wherein the auxiliary processing engine includes one or more crossbar switch matrices, each of the one or more crossbar switch matrices being configured to couple one or more image sensors to each of the two or more symmetric processing engines.

9. A method for processing data, the method comprising: Receiving input data by a symmetric processing engine among two or more symmetric processing engines coupled to an auxiliary processing engine; Receiving an indication to process the input data using a module of the auxiliary processing engine; Sending output data to the auxiliary processing engine; Receiving processed data from the auxiliary processing engine; Further processing the processed data in one or more parts of the pipeline of the modules of the symmetric processing engine to generate further processed data; And Outputting the further processed data.

10. The method according to claim 9, wherein the input data is received from two or more image sensors for processing by any one of the two or more symmetric processing engines.

11. The method according to claim 10, the method further comprising: Performing one or more data processing operations using less than all of the two or more image sensors.

12. The method according to claim 9, the method further comprising: Receiving output data from one of the two or more symmetric processing engines at a time.

13. The method according to claim 12, wherein the method further comprises: Switching between receiving output data from the symmetric processing engine and receiving output data from an additional symmetric processing engine among the two or more symmetric processing engines.

14. The method according to claim 9, the method further comprising: Receiving output data from the two or more symmetric processing engines concurrently.

15. The method according to claim 14, wherein the auxiliary processing engine includes a certain number of modules for performing data processing operations, and the number of the modules is based on the number of the two or more symmetric processing engines from which the auxiliary processing engine can concurrently receive output data.

16. The method according to claim 14, wherein the auxiliary processing engine includes one or more crossbar switch matrices, and each crossbar switch matrix in the one or more crossbar switch matrices is configured to couple one or more image sensors to each of the two or more symmetric processing engines.

17. An apparatus for processing data, the apparatus comprising: means for receiving input data through a symmetric processing engine among two or more symmetric processing engines coupled to an auxiliary processing engine; means for receiving an indication to process the input data using modules of the auxiliary processing engine; means for sending output data to the auxiliary processing engine; means for receiving processed data from the auxiliary processing engine; means for further processing the processed data in one or more parts of a pipeline of modules of the symmetric processing engine; and means for outputting the further processed data.

18. The apparatus according to claim 17, wherein the input data is received from two or more image sensors for processing by any one of the two or more symmetric processing engines.

19. The device according to claim 18, further comprising: Means for performing one or more data processing operations using less than all of the two or more image sensors.

20. The apparatus according to claim 17, wherein the apparatus further comprises: Means for receiving output data from one of the two or more symmetric processing engines at a time.

21. The device according to claim 20, wherein the device further comprises: Means for switching between receiving output data from the symmetric processing engine and receiving output data from an additional symmetric processing engine among the two or more symmetric processing engines.

22. The apparatus according to claim 17, wherein the apparatus further comprises: Means for receiving output data from the two or more symmetric processing engines concurrently.

23. The apparatus according to claim 22, wherein the auxiliary processing engine includes a certain number of modules for performing data processing operations, and the number of the modules is based on the number of the two or more symmetric processing engines from which the auxiliary processing engine can concurrently receive output data.

24. The apparatus according to claim 22, wherein the auxiliary processing engine includes one or more crossbar switch matrices, and each of the one or more crossbar switch matrices is configured to couple one or more image sensors to each of the two or more symmetric processing engines.

Images