Image processing resource sharing for image sensors

Abstract

An apparatus includes a first image processing path associated with a first image sensor and a second image processing path associated with a second image sensor. The apparatus also includes a multiplexer (MUX). The MUX is configured to provide data from the first image sensor to the second image processing path in accordance with a resource availability parameter associated with the second image processing path satisfying a resource availability condition.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 479,482, filed October 2, 2023, entitled “IMAGE PROCESSING RESOURCESHARING FOR IMAGE SENSORS,” which is expressly incorporated herein by reference in its entirety. Technical Field

[0002] All aspects of this disclosure relate to image processing in general, and more specifically to image processing resource sharing for image sensors. Background Technology

[0003] Image capture devices can capture one or more digital images (such as still images for photographs or sequences of images for video). Image capture devices can be integrated into a variety of devices. For example, image capture devices can be integrated into standalone digital cameras or digital video cameras, wireless communication devices equipped with cameras such as mobile phones (such as cellular phones, satellite radio phones, personal digital assistants (PDAs), and tablet devices), gaming devices, computing devices such as webcams and video surveillance cameras, and other devices with digital imaging or video capabilities.

[0004] Some image capture devices can generate relatively large amounts of data. For example, some image capture devices can generate image streams and data streams for autofocus (such as phase detection autofocus (PDAF) data). This type of data can be processed and buffered or stored in memory. As the amount of data increases, latency may occur. In some implementations (such as, for example, some extended reality (XR) implementations), such latency may be undesirable or impractical.

[0005] To reduce or avoid such latency, some image capture devices may increase power consumption, such as by using a "turbo" image processing mode. In some implementations (such as some battery-powered image capture devices), such an increase in power consumption may be undesirable or impractical. Summary of the Invention

[0006] The following summarizes some aspects of this disclosure to provide a basic understanding of the techniques discussed. This summary is not an exhaustive overview of all the intended features of this disclosure, nor is it intended to identify key or essential elements of all aspects of this disclosure, nor to define the scope of any or all aspects of this disclosure. Its purpose is to present some concepts of one or more aspects of this disclosure in a general form as a prelude to the more detailed description that follows.

[0007] In one aspect of this disclosure, an apparatus includes a first image processing path associated with a first image sensor, a second image processing path associated with a second image sensor, and a multiplexer (MUX). The MUX is configured to provide data from the first image sensor to the second image processing path based on resource availability parameters associated with the second image processing path to satisfy resource availability conditions.

[0008] In another aspect of this disclosure, an apparatus includes a first image processing path configured to receive an image signal associated with a first image sensor and generate image data based on the image signal. The apparatus also includes a second image processing path configured to receive an autofocus signal associated with a second image sensor and generate autofocus data based on the autofocus signal, provided that the resource availability parameter associated with the second image processing path satisfies the resource availability condition. The apparatus further includes a decoder configured to receive the image data from the first image processing path and the autofocus data from the second image processing path.

[0009] In another aspect of this disclosure, a method includes receiving a first image signal associated with a first image sensor at a first image processing path. The method further includes receiving data from the first image sensor at a second image processing path based on the satisfaction of a resource availability condition according to a resource availability parameter associated with a second image processing path. The second image processing path is associated with a second image sensor.

[0010] In another aspect of this disclosure, a method includes receiving an image signal associated with a first image sensor at a first image processing path. The method further includes generating image data based on the image signal, and receiving an autofocus signal associated with a second image sensor at a second image processing path based on the satisfaction of a resource availability condition according to a resource availability parameter associated with a second image processing path. The method also includes generating autofocus data based on the autofocus signal, and receiving the image data and the autofocus data at one or more shared image processing components associated with the first and second image processing paths.

[0011] The image processing methods described herein can be performed by an image capture device and / or on image data captured by one or more image capture devices. An image capture device (a device capable of capturing one or more digital images, whether still photographs or video sequences) can be incorporated into a variety of devices. By way of example, an image capture device may include a standalone digital camera or digital video camera, a wireless communication device equipped with a camera (such as a mobile phone, cellular, or satellite radio phone), a personal digital assistant (PDA), a panel or tablet device, a gaming device, a computing device (such as a webcam, video surveillance camera), or other devices with digital imaging or video capabilities.

[0012] The image processing techniques described herein can relate to a digital camera having an image sensor and processing circuitry (e.g., an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), or a central processing unit (CPU)). An image signal processor (ISP) may include one or more of these processing circuits and is configured to perform operations to acquire image data for processing according to the image processing techniques described herein and / or those involved in the image processing techniques described herein. An ISP may be configured to control the capture of image frames from one or more image sensors and to determine one or more image frames from said one or more image sensors to generate a view of a scene in an output image frame. The output image frame may be part of a sequence of image frames forming a video sequence. The video sequence may include additional image frames received from an image sensor or other image sensors.

[0013] In an example application, an image signal processor (ISP) may receive instructions for capturing a sequence of image frames in response to the loading of software, such as a camera application, to generate a preview display from an image capture device. The ISP may be configured to generate a single output image frame stream based on image frames received from one or more image sensors. The single output image frame stream may include raw image data from the image sensors, merged image data from the image sensors, or corrected image data processed by one or more algorithms within the ISP. For example, image frames may be processed by an image post-processing engine (IPE) and / or other image processing circuitry to process the image frames obtained from the image sensors (which may have undergone some processing before being output to the ISP), thereby performing one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, etc. The output image frames from the ISP may be stored in memory and retrieved by an application processor executing the camera application, which may perform further processing on the output image frames to adjust their appearance and reproduce them on a display for user viewing.

[0014] After an image signal processor and / or application processor (such as the image processing techniques described in the various embodiments herein) determines an output image frame representing a scene, the output image frame may be displayed on a device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, transmitted over a network, and / or printed to an output medium. For example, an image signal processor (ISP) may be configured to acquire input frames of image data (e.g., pixel values) from one or more image sensors and subsequently generate corresponding output image frames (e.g., preview display frames, still image captures, frames for video, frames for object tracking, etc.). In other examples, the image signal processor may output image frames to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., autofocus (AF), auto white balance (AWB), and auto exposure control (AEC)), to generate video files via the output frames, to configure frames for display, to configure frames for storage, to transmit frames via a network connection, etc. Generally, an image signal processor (ISP) can obtain incoming frames from one or more image sensors, generate an output frame stream, and output the output frame stream to various output destinations.

[0015] In some aspects, output image frames can be generated by combining various aspects of the image correction disclosed herein with other computational photographic techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, the first and second image frames are captured using different exposure times, different apertures, different lenses, and / or other characteristics that can result in improved dynamic range of the fused image when combining the two image frames. In some aspects, the method can be performed for MFNR photography, wherein the first and second image frames are captured using the same or different exposure times, and the first and second image frames are fused to generate a corrected first image frame that has reduced noise compared to the captured first image frame.

[0016] In some aspects, the device may include an image signal processor or processor (e.g., an application processor) that includes specific functionalities for camera control and / or processing, such as enabling or disabling the merging module or otherwise controlling aspects of image correction. The methods and techniques described herein may be performed entirely by the image signal processor or processor, or the various operations may be separated between the image signal processor and the processor, and in some aspects across additional processors.

[0017] The device may include one, two, or more image sensors, such as a first image sensor. When multiple image sensors are present, their configurations may differ. For example, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a long-range image sensor. In another example, the first sensor is configured to acquire an image through a first lens having a first optical axis, and the second sensor is configured to acquire an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. Any of these or other configurations may be part of a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located at offset positions on the front or rear of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image processing techniques described herein can be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0018] In an additional aspect of this disclosure, an apparatus configured for image processing and / or image capture is disclosed. The apparatus includes components for capturing image frames. The apparatus also includes one or more components for capturing data representing a scene, such as image sensors (including charge-coupled device (CCD), Bayer filter sensors, infrared (IR) detectors, ultraviolet (UV) detectors, complementary metal-oxide-semiconductor (CMOS) sensors) and time-of-flight detectors. The apparatus may further include components for focusing and / or directing light onto one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame input to the image processing techniques described herein.

[0019] Other aspects, features, and specific embodiments will become apparent to those skilled in the art when they review the following description of particular exemplary aspects in conjunction with the accompanying drawings. Although features may be discussed hereinafter with reference to certain aspects and drawings, various aspects may include one or more of the advantageous features discussed herein. In other words, while one or more aspects may be discussed having certain advantageous features, one or more such features may also be used depending on the various aspects. Similarly, although exemplary aspects may be discussed hereinafter as aspects of an apparatus, system, or method, exemplary aspects can be implemented in various apparatuses, systems, and methods.

[0020] This method can be embedded as computer program code in a computer-readable medium, the computer program code including instructions that cause a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device including: a first network adapter configured to transmit data, such as recorded images or videos or streaming data, via a first network connection among a plurality of network connections; and a processor coupled to the first network adapter and memory. The processor enables the output image frames described herein to be transmitted via a wireless communication network, such as a 5G NR communication network.

[0021] The features and technical advantages of the examples according to this disclosure have been summarized rather extensively above in order to better understand the detailed description below. Additional features and advantages will be described below. The disclosed concepts and specific examples can be readily used as the basis for modifying or designing other structures for achieving the same purpose of this disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein (both their organization and manner of operation) and the associated advantages will be better understood from the following description when considered in conjunction with the accompanying drawings. Each figure in the drawings is provided for illustration and description and not as a definition of limitation of the claims.

[0022] While aspects and implementations are described herein by way of example, those skilled in the art will understand that additional implementations and use cases may arise in many different arrangements and scenarios. The innovations described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and package arrangements. For example, aspects and / or devices may be implemented via integrated chip implementations and other devices based on non-modular components (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail / purchasing devices, medical devices, AI-enabled devices, etc.). While some examples may or may not specifically point to use cases or applications, applicability to various types of the described innovations is possible. The scope of implementations ranges from chip-level or modular components to non-modular, non-chip-level implementations, and further to aggregated, distributed, or original equipment manufacturer (OEM) devices or systems incorporating one or more aspects of the described innovations. In some practical settings, devices incorporating the described aspects and features may also necessarily include additional components and features for implementing and practicing the claimed and described aspects. For example, the transmission and reception of wireless signals necessarily involve multiple components (e.g., hardware components, including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.) for analog and digital purposes. The innovations described herein are intended to be implemented in a variety of devices, chip-level components, systems, distributed arrangements, end-user equipment, etc., with different sizes, shapes, and constructions. Attached Figure Description

[0023] Figure 1 This is an illustration of an example device for sharing image processing resources between image sensors, based on some aspects of this disclosure.

[0024] Figure 2 This is an illustration of another example device for sharing image processing resources between image sensors, based on some aspects of this disclosure.

[0025] Figure 3 This is an illustration of another example device for sharing image processing resources between image sensors, based on some aspects of this disclosure.

[0026] Figure 4 This is an illustration of another example device for sharing image processing resources between image sensors, based on some aspects of this disclosure.

[0027] Figure 5 This is a diagram illustrating an example data stream for sharing image processing resources between image sensors, according to some aspects of this disclosure.

[0028] Figure 6 This is a flowchart illustrating an example method for sharing image processing resources between image sensors according to some aspects of this disclosure.

[0029] Figure 7 This is a flowchart illustrating another example method for sharing image processing resources between image sensors, according to some aspects of this disclosure. Detailed Implementation

[0030] In some aspects of this disclosure, an apparatus may include a plurality of image sensors and a plurality of image processing paths associated with each of the plurality of image sensors. If one of the image sensors becomes idle, the apparatus may dynamically reassign the image processing path associated with the idle image sensor to a non-idle image sensor. In such examples, the plurality of image processing paths may process signals associated with the non-idle image sensors. For example, the signals may include image signals generated by the non-idle image sensors and autofocus signals (such as phase detection autofocus (PDAF) signals).

[0031] For further illustration, in some examples, the device may include a front-facing camera and a rear-facing camera. In one example, if the front-facing camera is in use, the device may reassign the image processing path associated with the rear-facing camera to the front-facing camera. In another example, if the rear-facing camera is in use, the device may reassign the image processing path associated with the front-facing camera to the rear-facing camera.

[0032] In some cases, dynamically reallocating resources from one image sensor to another can improve the performance of electronic devices. For example, by dynamically reallocating resources from one image sensor to another, a device can avoid initiating a "turbo" operating mode. Additionally, an idle (or "sleep") image sensor can remain in idle or sleep mode while sharing the image processing resources associated with that image sensor with another image sensor. Therefore, power consumption can be reduced without increasing latency.

[0033] The detailed description taken in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to limit the scope of this disclosure. Rather, the detailed description includes specific details for providing a thorough understanding of the subject matter of the invention. It will be apparent to those skilled in the art that these specific details are not necessary in every case, and in some cases, structures and components are shown in block diagram form for clarity of presentation.

[0034] In the description of the embodiments herein, numerous specific details (such as examples of specific components, circuits, and processes) are set forth to provide a thorough understanding of this disclosure. As used herein, the term "coupled" means a direct connection or a connection via one or more intermediate components or circuits. Moreover, specific terminology is set forth in the following description and for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that practicing the teachings disclosed herein may not require these specific details. In other instances, circuits and devices are shown in block diagram form to avoid obscuring the teachings of this disclosure.

[0035] Certain portions of the following detailed description are presented using other symbolic representations of programs, logic blocks, processes, and data bit operations within computer memory. In this disclosure, programs, logic blocks, processes, etc., are conceived as a self-consistent sequence of steps or instructions that produce a desired result. These steps are those that require physical manipulation of physical quantities. Although not strictly necessary, these physical quantities typically take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated within a computer system.

[0036] Example devices (such as smartphones) for capturing image frames using one or more image sensors may include a configuration of one, two, three, four, or more camera modules on the rear side (e.g., the side opposite the main user display) and / or the front side (e.g., the same side as the main user display). These devices may include one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., AI engines), or other suitable circuitry for processing the images captured by the image sensors. The one or more image signal processors (ISPs) may store the output image frames in memory (e.g., via a bus) and / or provide the output image frames to processing circuitry (e.g., an application processor). The processing circuitry may perform further processing, such as encoding, storing, transmitting, or other manipulation of the output image frames.

[0037] As used herein, a camera module may include an image sensor and certain other components coupled to the image sensor for acquiring a representation of a scene in image data comprising image frames. For example, a camera module may include other components of the camera, including a shutter, buffer, or additional readout circuitry for accessing individual pixels of the image sensor. In some embodiments, a camera module may include one or more components including an image sensor housed in a single package having an interface configured to couple the camera module to an image signal processor or other processor via a bus.

[0038] Figure 1 This is an illustration of an example device 100 for sharing image processing resources between image sensors according to some aspects of this disclosure. Device 100 may include or be otherwise coupled to one or more image signal processors (e.g., one or more ISPs 112) for processing image frames from one or more image sensors (such as a first image sensor 101, a second image sensor 102, and a depth sensor 140). In some specific embodiments, device 100 also includes or is coupled to processor 104 and memory 106 storing instructions 108 (e.g., memory storing processor-readable code or a non-transitory computer-readable medium storing instructions). Device 100 may also include or be coupled to display 114 and component 116. Component 116 may be used for user interaction, such as a touchscreen interface and / or physical buttons.

[0039] Component 116 may also include network interfaces for communicating with other devices, including a wide area network (WAN) adapter (e.g., WAN adapter 152), a local area network (LAN) adapter (e.g., LAN adapter 153), and / or a personal area network (PAN) adapter (e.g., PAN adapter 154). As an exemplary example, WAN adapter 152 may be a 4G LTE or 5G NR wireless network adapter. LAN adapter 153 may be an IEEE 802.11 WiFi wireless network adapter. PAN adapter 154 may be a Bluetooth wireless network adapter. Each of WAN adapter 152, LAN adapter 153, and / or PAN adapter 154 may be coupled to an antenna comprising multiple antennas configured for main and diversity reception and / or configured to receive a specific frequency band. In some embodiments, the antennas may be shared by WAN adapter 152, LAN adapter 153, and / or PAN adapter 154 for communication on different networks. In some implementations, WAN adapter 152, LAN adapter 153 and / or PAN adapter 154 may share circuitry and / or be packaged together, such as when LAN adapter 153 and PAN adapter 154 are packaged as a single integrated circuit (IC).

[0040] Device 100 may also include or be coupled to a power supply 118 for device 100, such as a battery or an adapter for coupling device 100 to an energy source. Device 100 may also include or be coupled to... Figure 1 Additional features or components not shown. In one example, a wireless interface that may include one or more transceivers and a baseband processor in a radio frequency front-end (RFFE) may be coupled to or may be included in the WAN adapter for use in a wireless communication device. In another example, an analog front-end (AFE) for converting analog image data to digital image data may be coupled between the first image sensor 101 or the second image sensor 102 and the processing circuitry in the device 100. In some embodiments, the AFE may be embedded in one or more ISPs 112.

[0041] The device may include or be coupled to a sensor hub 150, which interfaces with sensors to receive data about the movement of device 100, data about the environment surrounding device 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, a device configured to measure acceleration, which can also be used to determine the speed and distance of travel by appropriately integrating the measured acceleration. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub. In another example, the non-camera sensor may be a Global Positioning System (GPS) receiver, a device used to process satellite signals, such as through triangulation and other techniques, to determine the position of device 100. Position can be tracked over time to determine additional motion information, such as velocity and acceleration. Data from one or more sensors may be accumulated by the sensor hub 150 into motion data. One or more of acceleration, velocity, and / or distance may be included in motion data provided by sensor hub 150 to other components of device 100, including one or more ISPs 112 and / or processors 104.

[0042] One or more ISPs 112 can receive captured image data. In one embodiment, a local bus connection couples one or more ISPs 112 to a first image sensor 101 of a first camera 103 and a second image sensor 102 of a second camera 105, respectively. In another embodiment, a wired interface couples one or more ISPs 112 to an external image sensor. In yet another embodiment, a wireless interface couples one or more ISPs 112 to either the first image sensor 101 or the second image sensor 102.

[0043] First image sensor 101 and second image sensor 102 are configured to capture image data representing scenes within the fields of view of first camera 103 and second camera 105, respectively. In some embodiments, first camera 103 and / or second camera 105 output analog data converted by an analog front-end (AFE) and / or analog-to-digital converter (ADC) in device 100 or embedded in one or more ISPs 112. In some embodiments, first camera 103 and / or second camera 105 output digital data. The digital image data may be formatted into one or more image frames, whether received from first camera 103 and / or second camera 105 or converted from analog data received from first camera 103 and / or second camera 105.

[0044] The first camera 103 may include a first image sensor 101 and a first lens 131. The second camera may include a second image sensor 102 and a second lens 132. Each of the first lens 131 and the second lens 132 may be controlled by an associated autofocus (AF) algorithm (e.g., AF 133) executed in one or more ISPs 112, which adjusts the first lens 131 and the second lens 132 to focus on a specific focal plane located at a specific scene depth. AF 133 may be assisted by depth data received from a depth sensor 140. The first lens 131 and the second lens 132 focus light onto the first image sensor 101 and the second image sensor 102 respectively through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, and / or one or more color filter arrays (CFAs) for filtering light outside a specific frequency range. The first lens 131 and the second lens 132 may have different fields of view to capture different representations of the scene. For example, the first lens 131 may be an ultra-wide (UW) lens, and the second lens 132 may be a wide (W) lens. Multiple image sensors may include a combination of UW, W, long-range (T), and ultra-long-range (UT) sensors.

[0045] Each of the first camera 103 and the second camera 105 can be configured through hardware configuration and / or software settings to obtain different but overlapping fields of view. In some configurations, the cameras are configured with different lenses with different magnifications, resulting in different fields of view for capturing different representations of the scene. The cameras can be configured such that the UW camera has a larger FOV than the W camera, the W camera has a larger FOV than the T camera, and the T camera has a larger FOV than the UT camera. For example, a camera configured for a wide FOV can capture a field of view in the range of 64 to 84 degrees, a camera configured for an ultra-side FOV can capture a field of view in the range of 100 to 140 degrees, a camera configured for a long-range FOV can capture a field of view in the range of 10 to 30 degrees, and a camera configured for an ultra-long-range FOV can capture a field of view in the range of 1 to 8 degrees.

[0046] In some implementations, one or more of the first camera 103 and / or the second camera 105 may be a variable aperture (VA) camera, wherein the aperture can be adjusted to set a specific aperture size. Example aperture sizes include f / 2.0, f / 2.8, f / 3.2, f / 8.0, etc. Larger aperture values ​​correspond to smaller aperture sizes, and smaller aperture values ​​correspond to larger aperture sizes. The variable aperture (VA) camera may have different characteristics that produce different representations of the scene based on the current aperture size. For example, the VA camera may capture image data with a depth of focus (DOF) corresponding to the current aperture size set for the VA camera.

[0047] One or more ISPs 112 can process one or more image frames captured by the first camera 103, the second camera 105, or both. Although Figure 1 Device 100 is illustrated as including a first camera 103 and a second camera 105, but any number of cameras (e.g., one, two, three, four, five, six, etc.) may be coupled to one or more ISPs 112. In some aspects, a depth sensor (such as depth sensor 140) may be coupled to one or more ISPs 112. The output from depth sensor 140 may be processed in a manner similar to that of the first camera 103 and the second camera 105. Examples of depth sensors 140 include active sensors, including one or more of indirect time-of-flight (iToF), direct time-of-flight (dToF), mmWave, and / or hybrid depth sensors (such as structured light sensors). In embodiments without depth sensor 140, similar information about the depth or depth map of an object may be determined based on the parallax between the first camera 103 and the second camera 105, such as by using parallax depth measurement algorithms, stereo depth measurement algorithms, phase detection autofocus (PDAF) sensors, etc. Furthermore, any number of additional image sensors or image signal processors may be present for device 100.

[0048] In some embodiments, one or more ISPs 112 may execute instructions from memory, such as instructions 108 from memory 106, instructions stored in separate memory coupled to or included in one or more ISPs 112, or instructions provided by processor 104. Additionally or alternatively, one or more ISPs 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in this disclosure. For example, one or more ISPs 112 may include an image front-end (e.g., IFE 135), an image post-processing engine (e.g., IPE 136), an automatic exposure compensation (AEC) engine (e.g., AEC 134), and / or one or more engines for video analysis (e.g., EVA 137). The image pipeline may be formed by a sequence of one or more of IFE 135, IPE 136, and / or EVA 137. In some embodiments, the image pipeline can be reconfigured in one or more ISPs 112 by changing the connections between IFE 135, IPE 136, and / or EVA 137. AF 133, AEC 134, IFE 135, IPE 136 and EVA 137 may each include dedicated circuitry and may be embodied as software or firmware executed by one or more ISPs 112 and / or a combination of hardware and software or firmware executed on one or more ISPs 112.

[0049] Memory 106 may include a non-transient or non-transitory computer-readable medium storing computer-executable instructions (as instructions 108) for performing all or a portion of one or more of the operations described in this disclosure. Instructions 108 may include a camera application (or other suitable application, such as a messaging application) to be executed by device 100 for photographing or video recording. Instructions 108 may also include other applications or programs executed by device 100, such as an operating system and applications other than those for image or video generation. A camera application, such as one executed by processor 104, may enable device 100 to record images using a first camera 103 and / or a second camera 105 and one or more ISPs 112.

[0050] In addition to instruction 108, memory 106 may also store image frames. Image frames may be output image frames stored by one or more ISPs 112. Output image frames may be accessed by processor 104 for further operation. In some embodiments, device 100 does not include memory 106. For example, device 100 may be circuitry including one or more ISPs 112, and the memory may be external to device 100. Device 100 may be coupled to external memory and configured to access that memory to write output image frames for display or long-term storage. In some embodiments, device 100 is a system-on-a-chip (SoC) that integrates one or more ISPs 112, processor 104, sensor hub 150, memory 106, and / or component 116 into a single package.

[0051] In some embodiments, at least one of the one or more ISPs 112 or processor 104 executes instructions to perform, initiate, or control one or more operations described herein. For example, execution of instructions may instruct one or more ISPs 112 to begin or end the capture of image frames or sequences of image frames, wherein the capture includes corrections as described in the embodiments herein. In some embodiments, processor 104 may include one or more general-purpose processor cores 104A to 104N capable of executing instructions to control the operation of one or more ISPs 112. For example, cores 104A to 104N may execute a camera application (or other suitable application for generating images or videos) stored in memory 106 that activates or deactivates one or more ISPs 112 to capture image frames and / or controls one or more ISPs 112. The operation of cores 104A to 104N and one or more ISPs 112 may be based on user input. For example, a camera application executing on processor 104 may receive a user command to start a video preview display. Upon receiving the user command, video comprising a sequence of image frames is captured and processed from a first camera 103 and / or a second camera 105 via one or more ISPs 112 for display and / or storage. Image processing, such as that described herein, for determining “output” or “corrected” image frames, may be applied to one or more image frames in the sequence.

[0052] In some implementations, processor 104 may include one or more ICs or other hardware (e.g., an artificial intelligence (AI) engine, such as AI engine 124, or other coprocessors) to offload certain tasks from cores 104A through 104N. AI engine 124 may be used to offload tasks related to face detection and / or object recognition, performed, for example, using machine learning (ML) or artificial intelligence (AI). AI engine 124 may be referred to as an artificial intelligence processing unit (AI PU). AI engine 124 may include hardware configured to perform and accelerate convolutional operations involved in performing machine learning algorithms, such as by executing predictive models such as artificial neural networks (ANNs), including multilayer feedforward neural networks (MLFFNNs), recurrent neural networks (RNNs), and / or radial basis functions (RBFs). The ANN executed by AI engine 124 has access to predefined training weights for performing operations on user data. The ANN may optionally be trained during operation of image capture device 100, such as through reinforcement training, supervised training, and / or unsupervised training. In some other embodiments, device 100 does not include processor 104, such as when all the functionalities described are configured in one or more ISPs 112.

[0053] In some embodiments, display 114 may include one or more displays or screens that enable user interaction and / or present a preview of items (such as the output of the first camera 103 and / or the second camera 105) to the user. In some embodiments, display 114 is a touch-sensitive display. Input / output (I / O) components (such as component 116) may be or include any suitable mechanism, interface, or device to receive input (such as commands) from the user and provide output to the user via display 114. For example, component 116 may include (but is not limited to) a graphical user interface (GUI), keyboard, mouse, microphone, speaker, squeezable bezel, one or more buttons (such as a power button), slider, toggle key, switch, etc.

[0054] Although shown as coupled to each other via processor 104, components (such as processor 104, memory 106, one or more ISPs 112, display 114, and component 116) may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. An example of a bus used to interconnect components is a Peripheral Component Interface (PCI) Fast (PCIe) bus.

[0055] Although one or more ISPs 112 are illustrated as separate from processor 104, one or more ISPs 112 may be core of processor 104 (which is an application processor unit (APU)), included in a system-on-a-chip (SoC), or otherwise included in processor 104. While device 100 is referenced in the examples herein to perform aspects of this disclosure, some device components may not be... Figure 1 The details are shown to prevent obscuring aspects of this disclosure. Additionally, other components, the number of components, or combinations of components may be included in suitable equipment for performing aspects of this disclosure. Therefore, this disclosure is not limited to the configuration of a particular device or component, including device 100.

[0056] exist Figure 1 In the illustrated example, device 100 includes a first image processing path 141 associated with a first image sensor 101, and also includes a second image processing path 142 associated with a second image sensor 102. Figure 1The device 100 is also illustrated to include a multiplexer (MUX) 130. MUX 130 may include a first input coupled to a first image sensor 101, a second input coupled to a second image sensor 102, and an output coupled to a second image processing path 142. The first image processing path may be coupled to the first image sensor 101 and to one or more components (such as one or more ISPs 112) shared between the first image processing path 141 and the second image processing path 142. The second image processing path 142 may be coupled to MUX 130 and to one or more components (e.g., one or more ISPs 112).

[0057] In some aspects of this disclosure, device 100 may determine a resource availability parameter 144 associated with the second image processing path 142. For example, in some embodiments, one or more ISPs in ISP 112 may determine the resource availability parameter 144. In some examples, the resource availability parameter 144 may be based on a state associated with the second image sensor 102 or the second camera 105. For example, during operation of device 100, the second image sensor 102 may operate according to a first mode (such as an active mode, during which the second image sensor 102 captures one or more images) and a second mode (such as an idle mode, during which the second image sensor 102 does not capture one or more images).

[0058] Resource availability parameter 144 may be associated with a specific mode of the second image sensor 102. For example, a specific mode may include or correspond to a second mode (such as an idle mode). In some examples, resource availability parameter 144 may indicate whether the second image sensor 102 is associated with a first mode or a second mode, or it may indicate the amount of time the second image sensor 102 has been associated with a second mode. For example, if one or more ISPs 112 initiate (or are scheduled to initiate) an image capture operation using the second image sensor 102, one or more ISPs 112 may set resource availability parameter 144 to indicate the first mode. At other times, one or more ISPs 112 may set resource availability parameter 144 to indicate the second mode. For example, after the second image sensor 102 has completed an image capture operation, one or more ISPs 112 may set resource availability parameter 144 to indicate the second mode. In some examples, resource availability parameter 144 may include or be associated with a timestamp indicating the time when the second image sensor 102 was set to the second mode.

[0059] During operation, one or more ISPs 112 may monitor whether resource availability parameter 144 meets resource availability conditions (such as exceeding threshold 146). For example, resource availability parameter 144 may exceed threshold 146 based on the second image sensor 102 being idle for at least a specific time interval. In some examples, one or more ISPs 112 may use a timestamp that may be included in or associated with resource availability parameter 144 to determine that the second image sensor 102 has been associated with an idle operation mode for at least a specific time interval.

[0060] In some examples, resource availability parameter 144 may not exceed threshold 146. In such examples, device 100 may use MUX 130 to provide data from the second image sensor 102 to the second image processing path 142. For example, one or more ISPs 112 may set the control signal of MUX 130 to a first value (such as a logic "0" or logic "1") to couple the second image sensor 102 to the second image processing path 142 and cause MUX 130 to provide data from the second image sensor 102 to the second image processing path 142.

[0061] In some other examples, resource availability parameter 144 may exceed threshold 146. Based on resource availability parameter 144 exceeding threshold 146, device 100 may use MUX 130 to provide data from the first image sensor 101 to the second image processing path 142. For example, one or more ISPs 112 may set control signals of MUX 130 to a second value (such as a logic "1" or logic "0") to couple the first image sensor 101 to the second image processing path 142 and cause MUX 130 to provide data from the first image sensor 101 to the second image processing path 142. The data may include autofocus (AF) signals, such as phase detection autofocus (PDAF) signals.

[0062] Based on a change associated with resource availability parameter 144, device 100 can set a control signal to MUX 130 to a first value to couple the second image sensor 102 to the second image processing path 142. For example, if the second camera 105 is to initiate an image capture operation, the mode of the second image sensor 102 can be changed from idle mode to active mode. In such an example, device 100 can adjust the control signal from a second value to a first value to couple the second image sensor 102 to the second image processing path 142.

[0063] In some implementations, resource availability parameter 144 may correspond to or be associated with an operating mode of device 100. In some such examples, one or more ISPs 112 may determine whether resource availability parameter 144 satisfies a resource availability condition based at least in part on the operating mode of device 100 corresponding to a specific mode, or on the amount of time device 100 operates according to a mode satisfying a threshold amount of time. For illustration, in some examples, device 100 may include a front-facing camera (e.g., a first camera 103 or a second camera 105) and a rear-facing camera (e.g., a second camera 105 or a first camera 103). In one example, if the front-facing camera is in use, device 100 may determine that resource availability parameter 144 satisfies a resource availability condition and may reallocate an image processing path associated with the rear camera to the front-facing camera. In another example, if the rear camera is in use, device 100 may determine that resource availability parameter 144 satisfies a resource availability condition and may reallocate an image processing path associated with the front camera to the rear camera.

[0064] For further illustration, in some examples, one of the first camera 103 and the second camera 105 may be associated with head tracking, and the other of the first camera 103 and the second camera 105 may be associated with plane lookup and controller tracking. In some cases, head tracking may be enabled while (at least temporarily) plane lookup and controller tracking are disabled (or vice versa). In such examples, image processing resources associated with plane lookup and controller tracking may be allocated to head tracking (or vice versa). Other examples are also within the scope of this disclosure.

[0065] Figure 2 This is an illustration of an example device 200 for sharing image processing resources between image sensors according to some aspects of this disclosure. Device 200 may include reference... Figure 1 One or more features are described. For example, device 200 may include a first image sensor 101, a second image sensor 102, a MUX 130, a first image processing path 141, and a second image processing path 142. In some examples, device 200 is included Figure 1 In device 100 or corresponding Figure 1 100 devices.

[0066] exist Figure 2In some examples, the first image sensor 101 may include multiple encoders, such as a first encoder 208 and a second encoder 210. In some examples, the first encoder 208 and the second encoder 210 may each include or correspond to a Mobile Industrial Processor Interface (MIPI) encoder. In some other examples, as an illustrative example, one or both of the first encoder 208 or the second encoder 210 may each include or correspond to another type of encoder, such as an encoder for a standalone camera or a wearable device (such as a head-mounted device).

[0067] MUX 130 may include a control terminal 254. Control terminal 254 may be coupled to an input / output (I / O) pin 256 of device 200. In some examples, I / O pin 256 may correspond to a pin of an integrated circuit (such as a system-on-a-chip (SoC)). In some examples, I / O pin 256 may be a general-purpose input / output (GPIO) pin. MUX 130 may also be coupled to a first image sensor 101 (e.g., coupled to a second encoder 210) and to a second image sensor 102 (e.g., coupled to one or more encoders of the second image sensor 102).

[0068] Figure 2 The first image processing path 141 is also illustrated, which may include one or more of a physical camera interface 214, a decoder 218, and an ISP 222. The physical camera interface 214 may be coupled to a first image sensor 101 (e.g., coupled to a first encoder 208 and a second encoder 210) and may also be coupled to the decoder 218. The decoder 218 may be coupled to the physical camera interface 214 and to the ISP 222. The ISP 222 may be coupled to the decoder 218.

[0069] Additionally, the second image processing path 142 may include one or more of a physical camera interface 264, a decoder 268, and an ISP 272. The physical camera interface 264 may be coupled to the MUX 130 and the decoder 268. The decoder 268 may be coupled to the physical camera interface 264 and to the ISP 272. The ISP 272 may be coupled to the decoder 268. In some examples, one or both of the ISPs 222 and 272 may be included. Figure 1 One or more ISPs 112 may correspond to one or more ISPs.

[0070] During operation, the first image sensor 101 may generate a first image capture stream 202, and the second image sensor 102 may generate a second image capture stream 252. One or both of the first image capture stream or the second image capture stream 252 may include an image signal, an autofocus signal, or both or associated therewith. For example, the first image capture stream 202 may include an image signal 204 and an autofocus signal 206 or associated therewith. In some examples, the autofocus signal 206 may include or correspond to a PDAF signal.

[0071] The first encoder 208 can encode the image signal 204 to generate an encoded image signal 212. The second encoder 210 can encode the autofocus signal 206 to generate an encoded autofocus signal 230. The encoded image signal 212 and the encoded autofocus signal 230 may have a specific encoding format (such as MIPI encoding format).

[0072] The physical camera interface 214 can receive one or both of an encoded image signal 212 or an encoded autofocus signal 230. The physical camera interface 214 can perform analog-to-digital conversion on the signal from the first image sensor 101. For example, the physical camera interface 214 can perform analog-to-digital conversion on the encoded image signal 212 to generate image data 216.

[0073] Decoder 218 can receive image data from physical camera interface 214 and can decode the image data to generate decoded image data. For example, decoder 218 can receive image data 216 from physical camera interface 214 and can decode image data 216 to generate decoded image data 220.

[0074] The ISP 222 can receive decoded data from the decoder 218 and can perform image processing on the decoded data. For example, the ISP 222 can receive decoded image data 220 from the decoder 218 and can perform image processing on the decoded image data 220.

[0075] In some cases, the processing of both the encoded image signal 212 and the encoded autofocus signal 230 by the first image processing path 141 may be associated with relatively high latency, relatively high power consumption, or both. For illustration, in some specific implementations, the first image processing path 141 may process (or may be able to process) multiple signals in parallel with each other (e.g., by processing both the encoded image signal 212 and the encoded autofocus signal 230 in parallel with each other), thereby reducing or avoiding the latency associated with serial processing of such signals.

[0076] In some embodiments, parallel processing of the encoded image signal 212 and the encoded autofocus signal 230 by the first image processing path 141 can increase device power consumption, such as by increasing device power consumption compared to serial processing of such signals. For illustration, parallel processing of such signals can trigger device 200 to initiate a "turbo" operating mode, which increases power consumption to facilitate parallel processing. For example, in some embodiments, the encoded image signal 212 may be associated with a first clock frequency that differs from a second clock frequency associated with the encoded autofocus signal 230. To enable parallel processing of the encoded image signal 212 and the encoded autofocus signal 230, device 200 may initiate a turbo mode in which power consumption is increased. As a non-limiting illustrative example, in some embodiments, the encoded image signal 212 may be associated with a first clock frequency of 480 MHz, and the encoded autofocus signal 230 may be associated with a second clock frequency of 120 MHz. To increase the total clock frequency to 480MHz + 120MHz = 600MHz, device 200 can initiate a turbine operation mode, thereby increasing power consumption.

[0077] In some aspects of this disclosure, device 200 may perform image processing resource sharing to reduce or avoid initiating turbo operating modes. For example, MUX 130 may selectively couple a second image processing path 142 to either a first image sensor 101 (e.g., a second encoder 210) or a second image sensor 102. For instance, when the second image sensor 102 generates a second image capture stream 252, device 200 may set (e.g., via I / O pin 256) a first value for a control signal 255 provided to control terminal 254. Based on the first value of control signal 255, MUX 130 may couple a physical camera interface 264 to the second image sensor 102. In such examples, first image processing path 141 may process data associated with the first image capture stream 202 (such as encoded image signal 212 and encoded autofocus signal 230), and second image processing path 142 may process data associated with the second image capture stream 252 (such as one or both of the image signal or autofocus signal associated with the second image capture stream 252).

[0078] In some other examples, if the second image sensor 102 is not generating the second image capture stream 252, the device 200 may set a second value for the control signal 255 provided to the control terminal 254. Based on the second value of the control signal 255, the MUX 130 may couple the physical camera interface 264 to the first image sensor 101 (e.g., to the second encoder 210). In such examples, the first image processing path 141 may process the encoded image signal 212, and the second image processing path 142 may process the encoded autofocus signal 230.

[0079] For illustration, the second image processing path 142 may receive an encoded autofocus signal 262 and may perform analog-to-digital conversion on the encoded autofocus signal 262 using the physical camera interface 264 to generate autofocus data 266. The decoder 268 may receive the autofocus data 266 and may decode the autofocus data 266 to generate decoded autofocus data 270. The ISP 272 may receive the decoded autofocus data 270 and may perform image processing on the decoded autofocus data 270.

[0080] In some examples, the MUX 130 can be based on Figure 1 If the resource availability parameter 144 satisfies the resource availability condition (such as exceeding the threshold 146), data from the first image sensor 101 is provided to the second image processing path 142, as referenced. Figure 1 As described. As an illustrative example, ISP 272 (or another device, such as...) Figure 1 The processor 104 can monitor the operation of the second image sensor 102 and can set the value of the control signal 255 provided to the control terminal 254 according to the operating mode of the second image sensor 102. In some examples, the MUX 130 is configured to provide one of the encoded autofocus signal 230 or a signal associated with the second image capture stream 252 (e.g., an encoded image signal or an encoded autofocus signal associated with the second image capture stream 252) as data to the second image processing path 142. The MUX 130 can provide the signal associated with the second image capture stream 252 to the second image processing path 142 if the resource availability parameter 144 does not exceed the threshold 146.

[0081] One or more of ISP 222, ISP 272, or one or more other processors may perform image processing associated with the first image capture stream 202. For example, one or more of ISP 222, ISP 272, or one or more other processors may adjust the decoded image data 220 based on the decoded autofocus data 270. As an illustrative example, one or more of ISP 222, ISP 272, or one or more other processors may perform autofocus (AF), auto white balance (AWB), and auto exposure control (AEC) (3A) parameter synchronization on the decoded image data 220 based on the decoded autofocus data 270, may generate frame or video files based on the decoded image data 220 and the decoded autofocus data 270, and may configure one or more such frame or video files for display, storage, or transmission via a network connection, or combinations thereof, as an illustrative example.

[0082] Figure 3 This is an illustration of an example device 300 for sharing image processing resources between image sensors according to some aspects of this disclosure. Device 300 may include references Figure 1 and Figure 2 One or more features described in the document. For example, device 300 may include a first image sensor 101, a second image sensor 102, a MUX 130, a first image processing path 141, and a second image processing path 142. In some examples, device 300 is included Figure 1 In device 100 or corresponding Figure 1 100 devices. Figure 3 It is also illustrated that the first image processing path 141 may include a physical camera interface 214, and the second image processing path 142 may include a physical camera interface 264.

[0083] In addition, Figure 3 In one example, device 300 may include decoder 302 and ISP 304. Decoder 302 may be coupled to a first image processing path 141 (e.g., coupled to physical camera interface 214) and to a second image processing path 142 (e.g., coupled to physical camera interface 264). Decoder 302 may also be coupled to ISP 304. In some examples, decoder 302 may include a parallel connection to ISP 304. In some other examples, decoder 302 may include one or more serial connections to ISP 304. In some examples, ISP 304 may be included... Figure 1 One or more ISPs 112 or may correspond to one or more ISPs.

[0084] During operation, the MUX 130 can selectively couple the second image processing path 142 to the first image sensor 101 or to the second image sensor 102, such as a reference. Figure 1 and Figure 2 It is described by one or more of them. For example, the MUX130 may be based on Figure 1 Whether the resource availability parameter 144 meets the resource availability conditions (such as exceeding the threshold 146), the second image processing path 142 is selectively coupled to the first image sensor 101 or to the second image sensor 102.

[0085] Additionally, decoder 302 and ISP 304 can perform processing operations associated with both the first image sensor 101 and the second image sensor 102. For example, decoder 302 can receive image data 216 and decode the image data 216 to generate decoded image data 220. Decoder 302 can also receive autofocus data 266 and decode the autofocus data 266 to generate decoded autofocus data 270. ISP 304 can receive the decoded image data 220 and the decoded autofocus data 270 from decoder 302 and can perform image processing on the decoded image data 220 and the decoded autofocus data 270.

[0086] For further illustration, in some examples, the ISP 304 may adjust the decoded image data 220 based on the decoded autofocus data 270. As an illustrative example, the ISP 304 may perform autofocus (AF), auto white balance (AWB), and auto exposure control (AEC) (3A) parameter synchronization on the decoded image data 220 based on the decoded autofocus data 270, may generate frame or video files based on the decoded image data 220 and the decoded autofocus data 270, and may configure one or more such frame or video files for display, storage, or transmission via a network connection, or combinations thereof, as an illustrative example.

[0087] Figure 4 This is an illustration of an example device 400 for sharing image processing resources between image sensors according to some aspects of this disclosure. Device 400 may include references Figures 1 to 3 One or more of the features described in the document. For example, device 400 may include a first image sensor 101, a second image sensor 102, a first image processing path 141, and a second image processing path 142. In some examples, device 400 is included Figure 1 In device 100 or corresponding Figure 1 100 devices. Figure 4It is also illustrated that the first image processing path 141 may include a physical camera interface 214, and the second image processing path 142 may include a physical camera interface 264. The decoder 302 may be coupled to the first image processing path 141 (e.g., coupled to the physical camera interface 214) and to the second image processing path 142 (e.g., coupled to the physical camera interface 264). The decoder 302 may also be coupled to the ISP 304.

[0088] In addition, Figure 4 In some examples, the first image sensor 101 may include one or more encoders (including the first encoder 208), and the second image sensor 102 may include one or more encoders (including the second encoder 210). In some examples, the first image sensor 101 may correspond to the main image sensor of the device 400, and the second image sensor 102 may correspond to an auxiliary image sensor of the device 400. In some other examples, the first image sensor 101 may correspond to an auxiliary image sensor of the device 400, and the second image sensor 102 may correspond to the main image sensor of the device 400.

[0089] During operation, in at least some operating modes, the first image sensor 101 and the first image processing path 141 may collaboratively perform one or more operations with the second image sensor 102 and the second image processing path 142. For example, the device 400 may collaboratively perform such operations based on resource availability conditions (such as exceeding a threshold 146) met by a resource availability parameter 144. In some specific implementations, the first image sensor 101 and the first image processing path 141 may generate an image signal 204, an encoded image signal 212, and image data 216, and the second image sensor 102 and the second image processing path 142 may generate an autofocus signal 206, an encoded autofocus signal 262, and autofocus data 266. In some such examples, the second image sensor 102 and the second image processing path 142 may share idle image processing resources (e.g., by generating and processing the autofocus signal 206 on behalf of the first image sensor 101 and the first image processing path 141).

[0090] After generating and processing the autofocus signal 206 to generate autofocus data 266, the decoder 302 can receive image data 216 and autofocus data 266. The decoder 302 can decode the image data 216 to generate decoded image data 220, and can decode the autofocus data 266 to generate decoded autofocus data 270. The ISP 304 can receive the decoded image data 220 and the decoded autofocus data 270, and can perform image processing based on the decoded image data 220 and the decoded autofocus data 270. For example, the ISP 304 can generate a single image or a single image stream based on the decoded image data 220 and the decoded autofocus data 270.

[0091] In some examples, the second image sensor 102 and the second image processing path 142 may operate as auxiliary image processing paths based on one or more criteria. For example, if the second image sensor 102 is associated with a specific mode (such as an idle mode), the ISP 304 may reallocate (or “unload”) autofocus image processing operations from the first image sensor 101 and the first image processing path 141 to the second image sensor 102 and the second image processing path 142. For illustration, in some examples, the ISP 304 may perform such reallocation based on a resource availability parameter 144 exceeding a threshold 146. In such examples, if the second image sensor 102 generates additional image data associated with the autofocus signal 206 in the second image capture stream 252, the second image sensor 102 may discard the additional image data based on the image processing resources of the second image processing path 142 being allocated to the first image sensor 101 and the first image processing path 141. In other examples, the second image sensor 102 may avoid generating such additional image data by allocating image processing resources of the second image processing path 142 to the first image sensor 101 and the first image processing path 141.

[0092] In some other examples, the second image sensor 102 and the second image processing path 142 may not be available for resource sharing. In some such examples, the device 400 may perform separate image processing associated with the first image processing path 141 and the second image processing path 142. For example, if the device 400 detects that the second image sensor 102 is active (rather than idle), the device 400 may instruct the first image sensor 101 and the first image processing path 141 to perform both image capture and autofocus operations (e.g., rather than “offloading” such autofocus operations to the second image sensor 102 and the second image processing path 142).

[0093] Figure 5This is a diagram illustrating an example data stream 500 for sharing image processing resources between image sensors according to some aspects of this disclosure. It is described with reference to device 100. Figure 5 Data stream 500. Alternatively or otherwise, data stream 500 may be described with reference to one or more of devices 200, 300 and 400.

[0094] Figure 5 The example illustrates that device 100 can be implemented as a mobile device (such as a smartphone). In some other examples, device 100 can be implemented as another device (such as a standalone camera) or as a wearable device (such as a head-mounted device). Figure 5 In this device 100, a first surface 100a (e.g., a backward surface) and a second surface 100b (e.g., a forward surface) different from the first surface 100a may be positioned on the first surface 100a, and a second image sensor 102 may be positioned on the second surface 100b. In some other examples, the second image sensor 102 may be positioned on the first surface 100a, and the first image sensor 101 may be positioned on the second surface 100b.

[0095] During operation, processor 104 may communicate with one or more ISPs 112 (e.g., via a bidirectional bus and / or separate control and data lines). Processor 104 may control the first camera 103 and the second camera 105, for example, by executing camera control instructions 510. Camera control instructions 510 may include a camera driver executed by processor 104 for configuring the first camera 103 and the second camera 105, such as activating or deactivating image capture, configuring exposure settings, configuring aperture size, performing one or more other operations, or combinations thereof. Camera control instructions 510 may be managed by a camera application 504 executed by processor 104. Camera application 504 provides user-accessible settings, allowing a user to specify individual camera settings or select a profile with corresponding camera settings. Processor 104 may execute camera control instructions 510 to configure the first camera 103 and the second camera 105 according to commands received from camera application 504. Camera application 504 may be, for example, a photography application, a document scanning application, a messaging application, or other applications that process image data acquired from one or more of the first camera 103 or the second camera 105.

[0096] Camera configuration may include parameters specifying, such as frame rate, image resolution, readout duration, exposure level, aspect ratio, aperture size, etc. The first camera 103 or the second camera 105 may apply a corresponding camera configuration and may use that camera configuration to acquire image data representing the scene. In some embodiments, the camera configuration may be adjusted to obtain different representations of the scene. For example, the processor 104 may execute camera application 504 to instruct the first camera 103 to set a first camera configuration of the first camera 103 via camera control instructions 510, acquire first image data from the first camera 103 operating with the first camera configuration, instruct the first camera 103 to set a second camera configuration of the first camera 103, and acquire second image data from the first camera 103 operating with the second camera configuration. For example, the processor 104 can execute the camera application 504 to instruct the second camera 105 to set the third camera configuration of the second camera 105 via camera control instructions 510, obtain third image data from the second camera 105 operating in the third camera configuration, instruct the second camera 105 to set the fourth camera configuration of the second camera 105, and obtain fourth image data from the second camera 105 operating in the fourth camera configuration.

[0097] In some embodiments where the first camera 103 is a variable aperture (VA) camera system, the processor 104 can execute camera application 504 to instruct the first camera 103 to be configured to a first aperture size, acquire first image data from the first camera 103, instruct the first camera 103 to be configured to a second aperture size, and acquire second image data from the first camera 103. The aperture reconfiguration and the acquisition of the first and second image data can occur with little or no change in the scene captured at the first aperture size and the second aperture size. Example aperture sizes are f / 2.0, f / 2.8, f / 3.2, f / 8.0, etc. Larger aperture values ​​correspond to smaller aperture sizes, and smaller aperture values ​​correspond to larger aperture sizes. That is, f / 2.0 corresponds to an aperture size larger than f / 8.0.

[0098] Image data received from one or more of the first camera 103 or the second camera 105 can be processed in one or more blocks of one or more ISPs 112 to determine one or more output image frames 530 that can be stored in memory 106 and / or otherwise provided to processor 104. Processor 104 can further process the image data to apply effects to one or more output image frames 530. Effects may include background blur, lighting, color cast, and / or high dynamic range (HDR) blending. In some embodiments, one or more such effects may be applied in one or more ISPs 112 (e.g., as an alternative or supplement to the application of one or more such effects by processor 104).

[0099] Figure 6 This is a flowchart illustrating an example method 600 for sharing image processing resources between image sensors according to some aspects of this disclosure. In some examples, method 600 may be... Figure 1 Equipment 100 Figure 2 Equipment 200 Figure 3 The operation of method 600 may be performed by device 300 or another device. In some examples, as an illustrative example, one or more operations of method 600 may be initiated, performed or controlled by one or more processors, such as processor 104, one or more ISPs 112, ISP 222, ISP 272 or ISP 304.

[0100] At 602, method 600 includes receiving a first image signal associated with a first image sensor at a first image processing path. For example, the first image processing path 141 may receive an encoded image signal 212 (e.g., from a first encoder 208).

[0101] At 604, method 600 further includes determining whether a resource availability parameter satisfies a resource availability condition (such as threshold 146). For example, one or more processors may compare resource availability parameter 144 to threshold 146 to determine whether the resource availability parameter satisfies the resource availability condition. In some such examples, if resource availability parameter 144 exceeds threshold 146, the resource availability parameter satisfies the resource availability condition. In other such examples, if resource availability parameter 144 does not exceed threshold 146, the resource availability parameter may not satisfy the resource availability condition.

[0102] If the resource availability parameters cannot meet the resource availability conditions, method 600 may include setting a control signal to the MUX at 606 to a first value to couple the second image sensor to the second image processing path. For example, one or more processors may set a control signal 255 to the MUX 130 to a first value (such as a logic "0" or logic "1") to couple the second image sensor 102 to the second image processing path 142. Furthermore, at 610, method 600 may also include receiving a second image signal from the second image sensor at the second image processing path. For example, the second image processing path 142 may receive an encoded autofocus signal 262 from the second image sensor 102.

[0103] In some other examples, resource availability parameters may satisfy resource availability conditions. In such examples, method 600 may include adjusting a control signal to the MUX from a first value to a second value at 608 to couple the first image sensor to a second image processing path. For example, one or more processors may set a control signal 255 to the MUX 130 to a second value (such as a logic "1" or logic "0") to couple the first image sensor 101 to the second image processing path 142. Furthermore, at 612, method 600 may also include receiving data from the first image sensor at the second image processing path. For example, the second image processing path 142 may receive an encoded autofocus signal 230 from the first image sensor 101.

[0104] Figure 7 This is a flowchart illustrating an example method 700 for sharing image processing resources between image sensors according to some aspects of this disclosure. In some examples, method 700 may be... Figure 1 Equipment 100, by Figure 4 The operation of method 700 may be performed by device 400 or by another device. In some examples, as an illustrative example, one or more operations of method 700 may be initiated, performed, or controlled by one or more processors, such as processor 104, one or more ISPs 112, ISP 222, ISP 272, or ISP 304.

[0105] At 701, method 700 may include determining whether a resource availability parameter satisfies a resource availability condition (such as threshold 146). For example, one or more processors may compare resource availability parameter 144 to threshold 146 to determine whether the resource availability parameter satisfies the resource availability condition. In some such examples, if resource availability parameter 144 exceeds threshold 146, the resource availability parameter satisfies the resource availability condition. In other such examples, if resource availability parameter 144 does not exceed threshold 146, the resource availability parameter may not satisfy the resource availability condition.

[0106] If the resource availability parameter satisfies the resource availability condition, method 700 may further include receiving an image signal associated with the first image sensor at 702 in the first image processing path. For example, the first image processing path 141 may receive an encoded image signal 212 associated with the first image sensor 101.

[0107] At 704, method 700 further includes receiving an autofocus signal associated with a second image sensor at a second image processing path based on a resource availability condition satisfied by a resource availability parameter. For example, second image processing path 142 may receive an encoded autofocus signal 262 associated with second image sensor 102.

[0108] At 706, method 700 further includes generating image data based on the image signal. For example, physical camera interface 214 may generate image data 216 based on encoded image signal 212.

[0109] At 708, method 700 further includes generating autofocus data based on the autofocus signal. For example, physical camera interface 264 may generate autofocus data 266 based on encoded autofocus signal 262.

[0110] At 710, method 700 further includes receiving image data and autofocus data at one or more shared image processing components associated with the first image processing path and the second image processing path. For illustration, in some embodiments, the one or more shared image processing components may include decoders (such as decoder 302). Alternatively or in other embodiments, in some other embodiments, the one or more shared image processing components may include ISPs (such as one or more ISPs 112 or ISP 304).

[0111] In some other examples, resource availability parameters may not meet resource availability conditions. In some such examples, method 700 may also include performing separate image processing associated with the first image processing path and the second image processing path at 714. For example, if device 400 detects that the second image sensor 102 is active (rather than idle), device 400 may instruct the first image sensor 101 and the first image processing path 141 to perform both an image capture operation and an autofocus operation (e.g., rather than “offloading” such autofocus operations to the second image sensor 102 and the second image processing path 142).

[0112] While some examples have been provided for illustration, it should be understood that other examples are also within the scope of this disclosure. For example, although two image processing paths may be described in some embodiments, other embodiments may use a different number of image processing paths, such as three, four, or another number. Additionally, in some embodiments, the MUX 130 may be coupled to more than two image processing paths, such as three, four, or another number. In such examples, the MUX 130 may include more than two inputs, such as three, four, or another number. Furthermore, the control signal 255 may correspond to a multi-bit control signal that indicates a selected input from more than two inputs (such as three, four, or another number).

[0113] One or more components or devices described herein can be used in a variety of electronic devices. For example, as illustrative examples, the components and devices described herein can be implemented in mobile devices (such as cellular phones), stand-alone cameras, wearable devices (such as head-mounted displays), or other electronic devices. The features described herein can be used in one or more types of applications (including virtual reality (VR) applications, augmented reality (AR) applications, or extended reality (XR) applications). To further illustrate, in some examples, camera application 504 may correspond to VR, AR, or XR applications (such as games or simulations), and device 100 may correspond to wearable devices (such as VR, AR, or XR head-mounted displays). Other examples are also within the scope of this disclosure.

[0114] refer to Figures 1 to 7 One or more of the features described can improve the performance of an image capture device. For example, dynamically reallocating resources from one image sensor to another (such as from the first image sensor 101 to the second image sensor 102, or vice versa) can improve the performance of electronic devices (such as any of devices 100, 200, 300, and 400). To further illustrate, by dynamically reallocating resources from one image sensor to another, the device can avoid initiating a “turbo” operating mode. Additionally, an idle (or “sleep”) image sensor (such as one of the first image sensor 101 or the second image sensor 102) can remain in an idle or sleep mode, while the image processing resources associated with the idle or sleep image sensor are shared with the other image sensor (such as the other of the first image sensor 101 or the second image sensor 102). Therefore, power consumption can be reduced without increasing latency.

[0115] In a first aspect, an apparatus includes a first image processing path associated with a first image sensor, a second image processing path associated with a second image sensor, and a multiplexer (MUX). The MUX is configured to provide data from the first image sensor to the second image processing path based on resource availability parameters associated with the second image processing path, thereby satisfying resource availability conditions.

[0116] In a second aspect, in conjunction with the first aspect, the resource availability parameter is associated with a specific operating mode of the second image sensor.

[0117] In a third aspect, in conjunction with one or more of the first or second aspects, the particular operating mode includes an idle operating mode of the second image sensor, and the resource availability parameter satisfies the resource availability condition based on the second image sensor being associated with the idle operating mode for at least a specific time interval.

[0118] In a fourth aspect, in conjunction with one or more of the first to third aspects, the first image sensor includes a first encoder configured to generate an encoded image signal based on a first image capture stream associated with the first image sensor; and a second encoder configured to generate an encoded autofocus signal based on the first image capture stream. The MUX is further configured to provide either the encoded autofocus signal or a signal associated with the second image capture stream as data to the second image processing path.

[0119] In a fifth aspect, in conjunction with one or more of the first to fourth aspects, the second image sensor is configured to generate the second image capture stream, and the MUX is further configured to provide the signal associated with the second image capture stream to the second image processing path if the resource availability condition cannot be met based on the resource availability parameter.

[0120] In a sixth aspect, in conjunction with one or more of the first to fifth aspects, the first image sensor is positioned on a first surface of the device, and the second image sensor is positioned on a second surface of the device that is different from the first surface.

[0121] In a seventh aspect, in conjunction with one or more of the first to sixth aspects, the device also includes an input / output (I / O) pin coupled to a control terminal of the MUX.

[0122] In the eighth aspect, in conjunction with one or more of the first to seventh aspects, the first image processing path and the second image processing path each include one or more of a physical camera interface, a decoder, or an image signal processor (ISP).

[0123] In a ninth aspect, in conjunction with one or more of the first to eighth aspects, the device further includes a first physical camera interface for the first image processing path and a second physical camera interface for the second image processing path. The first physical camera interface is coupled to the first image sensor, and the second physical camera interface is coupled to the MUX. The device also includes a decoder coupled to both the first and second physical camera interfaces.

[0124] In the tenth aspect, in conjunction with one or more of the first to ninth aspects, the device also includes an image signal processor (ISP) coupled to the decoder.

[0125] In an eleventh aspect, an apparatus includes a first image processing path configured to receive an image signal associated with a first image sensor and generate image data based on the image signal. The apparatus also includes a second image processing path configured to receive an autofocus signal associated with a second image sensor based on a resource availability condition satisfied according to a resource availability parameter associated with the second image processing path, and to generate autofocus data based on the autofocus signal. The apparatus further includes a decoder configured to receive the image data from the first image processing path and the autofocus data from the second image processing path.

[0126] In the twelfth aspect, in conjunction with the eleventh aspect, the decoder is further configured to generate decoded image data based on the image data and to generate decoded autofocus data based on the autofocus data.

[0127] In the thirteenth aspect, in combination with one or more of the eleventh to twelfth aspects, the apparatus further includes an image signal processor (ISP) coupled to the decoder and configured to perform image processing based on the decoded image data and the decoded autofocus data.

[0128] In the fourteenth aspect, in conjunction with one or more of aspects eleven to thirteen, the first image sensor corresponds to the main image sensor of the device, and the second image sensor corresponds to the auxiliary image sensor of the device.

[0129] In the fifteenth aspect, in conjunction with one or more of aspects eleven to fourteen, the first image sensor is configured to generate a first image capture stream including the image data, the second image sensor is configured to generate a second image capture stream including the autofocus data, and the second image sensor is further configured to discard or avoid additional image data associated with the autofocus data in generating the second image capture stream.

[0130] In a sixteenth aspect, a method includes receiving a first image signal associated with a first image sensor at a first image processing path. The method further includes receiving data from the first image sensor at a second image processing path based on the satisfaction of a resource availability condition according to a resource availability parameter associated with a second image processing path. The second image processing path is associated with a second image sensor.

[0131] In the seventeenth aspect, in conjunction with the sixteenth aspect, the data is provided from the first image sensor to the second image processing path via a multiplexer (MUX).

[0132] In the eighteenth aspect, in conjunction with one or more of the sixteenth to seventeenth aspects, the method further includes adjusting a control signal to the MUX from a first value to a second value so that the MUX can couple the first image sensor to the second image processing path.

[0133] In a nineteenth aspect, in conjunction with one or more of aspects sixteen through eighteen, the method further includes setting the control signal to the first value based on a change associated with the resource availability parameter to couple the second image sensor to the second image processing path.

[0134] In a twentieth aspect, a method includes receiving an image signal associated with a first image sensor at a first image processing path. The method further includes generating image data based on the image signal, and receiving an autofocus signal associated with a second image sensor at a second image processing path based on the satisfaction of a resource availability condition according to a resource availability parameter associated with a second image processing path. The method further includes generating autofocus data based on the autofocus signal, and receiving the image data and the autofocus data at one or more shared image processing components associated with the first and second image processing paths.

[0135] In the twenty-first aspect, in conjunction with the twentieth aspect, the one or more shared image processing components include a decoder.

[0136] In the twentieth aspect, in combination with one or more of the twentieth to the twentieth-first aspects, the one or more shared image processing components include an image signal processor (ISP).

[0137] In the twenty-third aspect, in combination with one or more of the twenty to twenty-second aspects, the one or more shared image processing components include a decoder and an image signal processor (ISP).

[0138] In the accompanying drawings, a single block can be described as performing one or more functions. The one or more functions performed by this block can be performed in a single component or across multiple components, and / or can be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps are described below in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be construed as departing from the scope of this disclosure. Moreover, the example device may include components other than those shown, including components such as processors and memory.

[0139] The aspects of this disclosure are applicable to any electronic device that includes, is coupled to, or otherwise processes data from one, two, or more image sensors capable of capturing image frames (or “frames”). The terms “output image frame,” “modified image frame,” and “corrected image frame” can refer to an image frame that has been processed by any of the techniques disclosed to adjust the raw image data received from the image sensor. Additionally, aspects of the disclosed techniques can be implemented for processing image data received from image sensors having the same or different capabilities and characteristics, such as resolution, shutter speed, or sensor type. Furthermore, aspects of the disclosed techniques can be implemented in devices for processing image data, whether or not the device includes or is coupled to an image sensor. For example, the disclosed techniques may include operations performed by a processing device in a cloud computing system that retrieves image data previously recorded by a separate device having an image sensor for processing.

[0140] Unless explicitly stated otherwise in the following discussion, it should be understood that throughout this application, the use of terms such as “access,” “receive,” “transmit,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” “set,” “generate,” etc., refers to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the registers and memories of the computer system and transform them into other data similarly represented as physical quantities in the registers, memories, or other such information storage, transmission, or display devices of the computer system. The use of different terms to refer to actions or processes of a computer system does not necessarily indicate different operations. For example, “determining” data can refer to “generating” data. Similarly, “determining” data can refer to “retrieving” data.

[0141] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (such as a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that can implement at least some parts of this disclosure. Although the description and examples herein use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus can include a device or part of a device for performing the described operations.

[0142] Certain components in a device or apparatus described as “parts for access,” “parts for receiving,” “parts for transmitting,” “parts for using,” “parts for selecting,” “parts for determining,” “parts for normalizing,” “parts for multiplying,” or other similarly named terms referring to one or more operations on data (such as image data) may refer to processing circuitry (e.g., application-specific integrated circuit (ASIC), digital signal processor (DSP), graphics processing unit (GPU), central processing unit (CPU), computer vision processor (CVP), or neural signal processor (NSP)) configured to perform the described functions by means of hardware, software, or a combination of hardware configured by software.

[0143] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and skills. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0144] The components, functional blocks, and modules described herein with respect to the accompanying figures cited above include processors, electronic devices, hardware devices, electronic components, logic circuits, memory, software code, firmware code, and so on, or any combination thereof. Software should be interpreted broadly as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, and / or functions, whether referred to as software, firmware, middleware, microcode, hardware description languages, or other terms. Furthermore, the features discussed herein may be implemented via dedicated processor circuitry, via executable instructions, or a combination thereof.

[0145] Those skilled in the art will recognize that, although some features have been described individually for convenience, such features may be combined without departing from the scope of this disclosure. For illustration, see references to... Figures 1 to 7 The described one or more boxes (or operations) can be compared with the reference. Figures 1 to 7 Another diagram in the image describes a combination of one or more boxes (or operations). For example, in some aspects, the device may be based on... Figure 6 Method 600 and Figure 7 The method involves both of these methods.

[0146] Those skilled in the art will also recognize that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be construed as departing from the scope of this disclosure. Those skilled in the art will also readily recognize that the order or combination of components, methods, or interactions described herein are merely examples, and that components, methods, or interactions of various aspects of this disclosure can be combined or performed in ways other than those illustrated and described herein.

[0147] The various exemplary logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various exemplary components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0148] Hardware and data processing means for implementing the various exemplary logic units, logic blocks, modules, and circuits described herein can be implemented or executed using general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic units, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor can be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some embodiments, the processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some embodiments, specific processes and methods may be performed by circuitry specific to a given function.

[0149] In one or more aspects, the described functionality may be implemented in hardware, digital electronic circuits, computer software, firmware, including the structures disclosed in this specification and their structural equivalents or any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus.

[0150] If implemented in software, the functionality may be stored as one or more instructions or code on or transmitted through a computer-readable medium. The processes of the methods or algorithms disclosed herein may be implemented in a processor-executable software module that may reside on a computer-readable medium. Computer-readable media include computer storage media. Storage media may be any available medium accessible to a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible to a computer. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically reproduce data, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operation of a method or algorithm may reside as one of code and instructions, or any combination or set of code and instructions, on a machine-readable medium and a computer-readable medium that may be incorporated into a computer program product.

[0151] Various modifications to the specific embodiments described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific embodiments without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the specific embodiments shown herein, but are to be granted the broadest scope consistent with this disclosure, the principles disclosed herein, and the novel features.

[0152] Additionally, those skilled in the art will readily recognize that, for the convenience of describing the accompanying drawings, contrasting terms such as “upper” and “lower” or “front” and “back” or “top” and “bottom” or “forward” and “backward” are sometimes used, indicating relative positions on a correctly oriented page corresponding to the orientation of the drawings, and may not reflect the correct orientation of any device as implemented.

[0153] Certain features described in this specification in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations and even originally claimed in this way, one or more features from the claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0154] Similarly, although operations are depicted in a specific order in the figures, this should not be construed as requiring such operations to be performed in the shown specific order or sequential order, or to perform all illustrated operations to achieve the desired result. Additionally, the figures may schematically depict one or more example processes in the form of flowcharts. However, other operations not depicted may be combined with the schematically illustrated example processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any illustrated operation. In some contexts, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other embodiments also fall within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result.

[0155] As used herein (including the claims), the term "or" in a list of two or more items means that any one of the listed items may be used alone, or any combination of two or more listed items may be used. For example, if a composition is described as containing component A, B, or C, the composition may contain A alone; B alone; C alone; a combination of A and B; a combination of A and C; a combination of B and C; or a combination of A, B, and C. Moreover, as used herein (including the claims), "or" in a list of items beginning with "at least one of" indicates a separate list, such that a list such as "at least one of A, B, or C" refers to A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these items.

[0156] The term “substantially” is defined as being largely but not necessarily entirely what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any specific implementation of the disclosure, the term “substantially” may be used in place of the “[percentage]” of the specified content, where the percentage includes 0.1%, 1%, 5%, or 10%.

[0157] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An apparatus, the apparatus comprising: The first image processing path associated with the first image sensor; A second image processing path associated with the second image sensor; and A multiplexer (MUX) is configured to provide data from the first image sensor to the second image processing path based on resource availability conditions satisfied by resource availability parameters associated with the second image processing path.

2. The apparatus of claim 1, wherein the resource availability parameter is associated with a specific operating mode of the second image sensor.

3. The apparatus of claim 2, wherein the specific operating mode includes an idle operating mode of the second image sensor, and wherein the resource availability parameter satisfies the resource availability condition based on the second image sensor being associated with the idle operating mode for at least a specific time interval.

4. The apparatus of claim 1, wherein the first image sensor comprises: A first encoder, configured to generate an encoded image signal based on a first image capture stream associated with the first image sensor; and A second encoder, configured to generate an encoded autofocus signal based on the first image capture stream, The MUX is further configured to provide either the encoded autofocus signal or a signal associated with the second image capture stream as data to the second image processing path.

5. The apparatus of claim 4, wherein the second image sensor is configured to generate the second image capture stream, and wherein the MUX is further configured to provide the signal associated with the second image capture stream to the second image processing path if the resource availability condition cannot be met according to the resource availability parameter.

6. The apparatus of claim 1, wherein the first image sensor is positioned on a first surface of the apparatus, and wherein the second image sensor is positioned on a second surface of the apparatus different from the first surface.

7. The apparatus of claim 1, further comprising an input / output (I / O) pin, wherein the I / O pin is coupled to a control terminal of the MUX.

8. The apparatus of claim 1, wherein the first image processing path and the second image processing path each comprise one or more of a physical camera interface, a decoder, or an image signal processor (ISP).

9. The apparatus according to claim 1, further comprising: The first physical camera interface of the first image processing path is coupled to the first image sensor. The second physical camera interface of the second image processing path is coupled to the MUX; and A decoder, which is coupled to the first physical camera interface and the second physical camera interface.

10. The apparatus of claim 9, further comprising an image signal processor (ISP) coupled to the decoder.

11. An apparatus comprising: A first image processing path is configured to receive an image signal associated with a first image sensor and generate image data based on the image signal. A second image processing path is configured to receive an autofocus signal associated with a second image sensor based on a resource availability condition satisfied according to a resource availability parameter associated with the second image processing path, and to generate autofocus data based on the autofocus signal. and A decoder configured to receive the image data from the first image processing path and the autofocus data from the second image processing path.

12. The apparatus of claim 11, wherein the decoder is further configured to generate decoded image data based on the image data and to generate decoded autofocus data based on the autofocus data.

13. The apparatus of claim 12, further comprising an image signal processor (ISP) coupled to the decoder and configured to perform image processing based on the decoded image data and the decoded autofocus data.

14. The apparatus of claim 11, wherein the first image sensor corresponds to the main image sensor of the apparatus, and wherein the second image sensor corresponds to the auxiliary image sensor of the apparatus.

15. The apparatus of claim 11, wherein the first image sensor is configured to generate a first image capture stream including the image data, wherein the second image sensor is configured to generate a second image capture stream including the autofocus data, and wherein the second image sensor is further configured to discard or avoid additional image data associated with the autofocus data in generating the second image capture stream.

16. A method comprising: At the first image processing path, a first image signal associated with the first image sensor is received; as well as Based on the resource availability parameter associated with the second image processing path satisfying the resource availability condition, data from the first image sensor is received at the second image processing path, which is associated with the second image sensor.

17. The method of claim 16, wherein the data is provided from the first image sensor to the second image processing path via a multiplexer (MUX).

18. The method of claim 17, further comprising adjusting a control signal to the MUX from a first value to a second value, such that the MUX can couple the first image sensor to the second image processing path.

19. The method of claim 18, further comprising setting the control signal to the first value based on a change associated with the resource availability parameter to couple the second image sensor to the second image processing path.

20. A method comprising: At the first image processing path, an image signal associated with the first image sensor is received; Image data is generated based on the image signal; Based on the resource availability parameters associated with the second image processing path satisfying the resource availability conditions, an autofocus signal associated with the second image sensor is received at the second image processing path; Automatic focus data is generated based on the autofocus signal; as well as The image data and the autofocus data are received at one or more shared image processing components associated with the first image processing path and the second image processing path.

21. The method of claim 20, wherein the one or more shared image processing components include a decoder.

22. The method of claim 20, wherein the one or more shared image processing components include an image signal processor (ISP).

23. The method of claim 20, wherein the one or more shared image processing components include a decoder and an image signal processor (ISP).

Images