Systems and methods for multi-context image capture

Abstract

Imaging systems and techniques are described. For example, an imaging system may receive raw image data captured using an image sensor. The imaging system may process the raw image data in accordance with a first image signal processor (ISP) setting to generate a first image and process the raw image data in accordance with a second ISP setting that is different from the first ISP setting to generate a second image. The imaging system may output the first image and the second image.

Description

Technical Field

[0001] The present disclosure generally relates to imaging. For example, aspects of the present disclosure relate to systems and techniques for multi-context image capture. Background Art

[0002] A camera is a device that uses an image sensor to receive light and capture image frames, such as still images or video frames. A camera may include a processor, such as an image signal processor (ISP), that may receive one or more image frames and process the one or more image frames. For example, raw image frames captured by a camera sensor may be processed by an ISP to generate a final image. A camera may be configured with various image capture and image processing settings to change the appearance of an image. Some camera settings are determined and applied before or during the ISP capture of a photo. Other camera settings may configure post-processing of a photo. Traditionally, ISP settings are set during camera calibration (e.g., during camera manufacturing) and are not changed thereafter. However, in some cases, different ISP settings may be better than other ISP settings in different contexts and / or for different purposes. Summary of the invention

[0003] Systems and techniques for multi-context image capture are described herein. According to various aspects described herein, these systems and techniques can perform imaging in which different imaging settings are selected and used for image capture for different contexts, such as images intended for display to a human viewer and images intended for computer vision applications.

[0004] According to at least one example, a device for multi-context image capture is provided. The device includes a memory and at least one processor coupled to the memory (e.g., implemented in a circuit). The at least one processor is configured and capable of: receiving raw image data captured using an image sensor; processing the raw image data according to a first image signal processor (ISP) setting to generate a first image; processing the raw image data according to a second ISP setting to generate a second image, wherein the second ISP setting is different from the first ISP setting; and outputting the first image and the second image.

[0005] In another example, a method of multi-context image capture is provided. The method includes: receiving raw image data captured using an image sensor; using an image signal processor (ISP), processing the raw image data according to a first ISP setting to generate a first image; using the ISP, processing the raw image data according to a second ISP setting to generate a second image, wherein the second ISP setting is different from the first ISP setting; and outputting the first image and the second image.

[0006] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: receive raw image data captured using an image sensor; process the raw image data according to a first image signal processor (ISP) setting to generate a first image; process the raw image data according to a second ISP setting to generate a second image, wherein the second ISP setting is different from the first ISP setting; and output the first image and the second image.

[0007] In another example, an apparatus for multi-context image capture is provided. The apparatus includes: a component for receiving raw image data captured using an image sensor; a component for processing the raw image data according to a first image signal processor (ISP) setting to generate a first image; a component for processing the raw image data according to a second ISP setting to generate a second image, wherein the second ISP setting is different from the first ISP setting; and a component for outputting the first image and the second image.

[0008] In some aspects, the apparatus is part of and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smart phone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or a component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary aspects of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 is a block diagram illustrating an example architecture of an image capture and processing system according to some examples;

[0013] Figure 2 is a block diagram illustrating an example architecture for a multi-context imaging process performed using an imaging system according to some examples;

[0014] Figure 3A is a perspective view illustrating a head mounted display (HMD) used as part of an imaging system according to some examples;

[0015] Figure 3B is an example based on some examples Figure 3A A perspective view of a head mounted display (HMD) being worn by a user;

[0016] Figure 4A is a perspective view illustrating a front surface of a mobile phone including a front-facing camera and usable as part of an imaging system according to some examples;

[0017] Figure 4B is a perspective view illustrating a rear surface of a mobile phone including a rear camera and usable as part of an imaging system according to some examples;

[0018] Figure 5 is a perspective view illustrating a vehicle including various sensors according to some examples;

[0019] Figure 6 is a block diagram illustrating a row-interleaved multi-sensor multi-context imaging process performed using an imaging system according to some examples;

[0020] Figure 7 is a block diagram illustrating a frame-interleaved multi-context imaging process performed using an imaging system according to some examples;

[0021] Figure 8 is a block diagram illustrating an example architecture for a multi-context imaging process performed using an imaging system according to some examples;

[0022] Fig. 9 is a block diagram illustrating an example of a neural network that may be used for imaging operations according to some examples;

[0023] Fig.10 is a flow diagram illustrating an imaging process according to some examples; and

[0024] Fig.11 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0025] Certain aspects of the present disclosure are provided below. Some of these aspects may be applied independently, and some of them may be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. Each of the drawings and descriptions is not intended to be restrictive.

[0026] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0027] A camera is a device that uses an image sensor to receive light and capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera may be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, aperture value, shutter speed, focus, and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters may be applied to an image sensor used to capture one or more image frames. Other camera settings may configure post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, levels, curves, or colors. For example, settings or parameters may be applied to a processor (e.g., an image signal processor or ISP) used to process one or more image frames captured by an image sensor.

[0028] Traditionally, ISP settings are set during camera calibration (e.g., during camera manufacturing) and are not changed thereafter. For example, conventional solutions use one ISP for each context or each sensor. However, in some cases, different ISP settings may be better than other ISP settings in different contexts and / or for different purposes. For example, one set of ISP settings may be optimized to produce images that are clear, sharp, realistic, and / or color correct based on human vision of one or more human viewers. Another set of ISP settings may be optimized to produce images that are useful for machine learning and / or computer vision algorithms (such as algorithms for object detection, object recognition, and / or object tracking). In some cases, aspects such as white balance, tone mapping, color mapping, chromatic aberration correction (CAC), and lens distortion correction may be more important for human vision than for computer vision. Which settings are best for computer vision may depend on the algorithms used and / or the training of the trained machine learning models used.

[0029] Imaging systems and techniques are described. An imaging system receives raw image data captured using an image sensor or multiple image sensors. The imaging system generates a first image using the raw image data at least in part by processing the raw image data according to a first ISP setting using an image signal processor (ISP). The imaging system generates a second image using the raw image data at least in part by processing the raw image data according to a second ISP setting different from the first ISP setting using the ISP. In some examples, the first ISP setting is configured for human vision and the second ISP setting is configured for computer vision. The imaging system outputs the first image and the second image, for example, by displaying the first image and feeding the second image to a machine learning model trained for computer vision.

[0030] The imaging systems and techniques described herein provide many technical improvements over existing imaging systems. For example, the imaging systems and techniques described herein allow the same raw image data to be used to capture images optimized for different contexts. For example, in the example of a camera of a vehicle, the raw image data captured by the vehicle may be captured as a first image according to a first set of ISP settings to show the driver a realistic-looking image of the environment, but may also be captured as a second image according to a second set of ISP settings to be passed to a computer vision algorithm, thereby detecting pedestrians, vehicles, and / or possible routes for vehicles based on the image. The first image may have accurate color and brightness levels, reduced lens distortion, etc. The second image may be optimized in terms of edge detection and contrast of certain types of features to maximize the probability of accurately detecting an object. The imaging systems and techniques described herein provide row-level interleaving or frame-level interleaving of the imaging process performed by the ISP, thereby allowing image capture to be performed sequentially, in parallel, or a combination thereof. By at least these means, the imaging systems and techniques described herein provide efficient capture of the same raw image data according to different ISP settings for different contexts, and improve both the usability of human users and the functionality of computer vision systems.

[0031] Various aspects of the application will be described with respect to the accompanying drawings. Figure 1 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of one or more scenes (e.g., images of scene 110). The image capture and processing system 100 may capture independent images (or photographs) and / or may capture a video including multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130. In some examples, the scene 110 is a scene in an environment. In some examples, the scene 110 is a scene of at least a portion of a user. For example, the scene 110 may be a scene of one or both of the user's eyes and / or at least a portion of the user's face.

[0032] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include multiple mechanisms and components; for example, the control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms other than those illustrated, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0033] A focus control mechanism 125B of the control mechanism 120 may obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B may adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may move the lens 115 closer to the image sensor 130 or further away from the image sensor 130 by actuating a motor or a servo, thereby adjusting the focus. In some cases, an additional lens may be included in the system 100, such as one or more micro lenses above each photodiode of the image sensor 130, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined by contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0034] The exposure control mechanism 125A of the control mechanism 120 may obtain an exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or aperture value), the duration of the aperture opening (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0035] A zoom control mechanism 125C of the control mechanism 120 may obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos to move one or more lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (which may be the lens 115 in some cases) that first receives light from the scene 110, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. The afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses having equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0036] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image generated by the image sensor 130. In some cases, different photodiodes may be covered by different color filters and thus may measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of an image is generated based on red light data from at least one photodiode covered in a red filter, blue light data from at least one photodiode covered in a blue filter, and green light data from at least one photodiode covered in a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald") filters instead of or in addition to red, blue, and / or green filters. Some image sensors may have no color filters at all and may instead use different photodiodes (in some cases vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light.Monochrome image sensors may also lack color filters, and therefore lack color depth.

[0037] In some cases, image sensor 130 may alternatively or additionally include an opaque mask and / or a reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying an analog signal output by a photodiode and / or an analog-to-digital converter (ADC) for converting an analog signal output by a photodiode (and / or an analog signal amplified by an analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0038] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more processors of any other type of processor 1110 discussed with respect to the computing system 1100. The host processor 152 may be a digital signal processor (DSP) and / or other type of processor. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM, global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0039] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1120, read-only memory (ROM) 145 and / or 1125, cache, memory unit, another storage device, or some combination thereof.

[0040] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touch pad, a touch-sensitive surface, a printer, any other output device 1135, any other input device 1145, or some combination thereof. In some cases, subtitles may be input into the image processing device 105B through a physical keyboard or keypad of the I / O device 160, or through a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that enable a wireless connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0041] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0042] like Figure 1 As shown, the vertical dotted line will Figure 1 1 is divided into two parts, representing image capture device 105A and image processing device 105B. Image capture device 105A includes lens 115, control mechanism 120 and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145 and I / O 160. In some cases, some components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0043] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communications (such as cellular network communications, 1102.11 wi-fi communications, wireless local area network (WLAN) communications, or some combination thereof). In some specific implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing devices.

[0044] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0045] Figure 2is a block diagram illustrating an example architecture of a multi-context imaging process performed using an imaging system 200. The imaging system 200 may include at least one of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the HMD 310, the mobile phone 410, the vehicle 510, the imaging system 600, the imaging system 700, the imaging system 800, the neural network 900, the imaging system performing the imaging process 1000, the computing system 1100, or a combination thereof. In some examples, the imaging system 200 may include, for example, one or more laptops, phones, tablet computers, mobile phones, video game consoles, vehicle computers, desktop computers, wearable devices, televisions, media centers, XR systems, head mounted display (HMD) devices, other types of computing devices discussed herein, or a combination thereof.

[0046] The imaging system 200 includes one or more sensors 205 configured to capture raw sensor data 210. In some examples, the sensor 205 includes one or more image sensors or one or more cameras. In some aspects, the sensor 205 includes a plurality of sensors. In some cases, each of the plurality of sensors may be asynchronous relative to at least one other of the plurality of sensors (e.g., a first sensor of the plurality of sensors is asynchronous relative to a second sensor, a third sensor, etc.), or all of the sensors may be asynchronous relative to each other. In some cases, at least two of the plurality of sensors may be synchronized relative to each other. In some examples, the frame rate and / or resolution of each of the plurality of sensors may be different from at least one other of the plurality of sensors, or all of the sensors may have different frame rates and / or resolutions.

[0047] In some examples, raw sensor data 210 captured using sensors 205 includes raw image data, image data, pixel data, image frames, raw video data, video data, video frames, or a combination thereof. In some examples, at least one of sensors 205 may be oriented toward a user (e.g., may be facing the user), and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of the user. In some examples, at least one of sensors 205 may be facing away from the user (e.g., may be facing away from the user) and / or toward an environment in which the user is located, and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of the environment. In some examples, sensor data captured by at least one of sensors 205 that is facing away from the user and / or toward may have a field of view (FoV) that includes, is included in, overlaps with, and / or otherwise corresponds to the FoV of the user's eyes. Figure 2 In FIG. 1 , a graphic representing sensor 205 illustrates sensor 205 as including a camera and a microphone facing an environment in which two persons are located. Figure 2 Within, a graph representing raw sensor data 210 illustrates a representation of an environment illustrated for sensor 205 divided into data corresponding to different pixels and / or photodiodes of sensor 205 .

[0048] In some examples, the sensors 205 may include one or more cameras, image sensors, microphones, heart rate monitors, oximeters, biometric sensors, positioning receivers, global navigation satellite system (GNSS) receivers, inertial measurement units (IMUs), accelerometers, gyroscopes, gyrometers, barometers, thermometers, altimeters, depth sensors, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, time of flight (ToF) sensors, structured light sensors, other sensors discussed herein, or combinations thereof. In some examples, one or more sensors 205 include at least one image capture and processing system 100, image capture device 105A, image processing device 105B, or combinations thereof. In some examples, one or more sensors 205 include at least one input device 1145 of computing system 1100. In some implementations, one or more of the sensors 205 may supplement or refine sensor readings from other sensors 205. For example, an inertial measurement unit (IMU), accelerometer, gyroscope, or other sensor may be used to identify the pose (e.g., position and / or orientation) of the imaging system 200 and / or the user in the environment, and / or the user's gaze through the imaging system 200.

[0049] In some examples, sensor 205 captures sensor data that measures and / or tracks information about various aspects of a user's body and / or the user's behavior (e.g., characteristics of the user). In some examples, sensor 205 includes one or more image sensors of one or more cameras facing at least a portion of a user (e.g., at least a portion of a user's face and / or head). The one or more cameras may include one or more image sensors that capture image data, which includes one or more images of at least a portion of the user. For example, sensor 205 may include one or more image sensors focused on one or both eyes (and / or eyelids) of a user, wherein the image sensors of the cameras capture image data of one or both eyes of the user. One or more image sensors may also be referred to as eye capture sensors. In some implementations, one or more image sensors may capture image data including a series of images over time, which in some examples may be arranged together in time order, such as to form a video. The series of images may depict or otherwise indicate, for example, movement of the user's eyes, pupil dilation, blinking (using eyelids), squinting (using eyelids), saccades, gazes, eye moisture levels, optokinetic reflexes or reactions, vestibulo-ocular reflexes or reactions, accommodation reflexes or reactions, other properties related to the eyes and / or eyelids described herein, or combinations thereof. Sensors 205 may include one or more sensors that track information about the user and / or environment, including the user's posture (e.g., position and / or orientation) and / or the user's behavior.

[0050] The imaging system 200 includes a sensor signal processor 220 configured to receive raw sensor data 210 from the sensor 205. The sensor signal processor 220 processes the raw sensor data 210 to generate first sensor data 230 and second sensor data 235. In some examples, the sensor signal processor 220 includes an image signal processor (ISP), such as the ISP 154. In some examples, the sensor signal processor 220 includes the image processor 150, the host processor 152, the ISP 154, the processor 1110, or a combination thereof. In some examples, the sensor signal processor 220 processes the raw sensor data 210 according to various ISP settings that tune various ISP parameters. In some examples, the ISP parameters may include brightness, color balance, tone mapping, sharpness, noise reduction, saturation, white balance, or a combination thereof. Different ISP settings for these ISP parameters may adjust the following: the sensor signal processor 220 applies these ISP parameters to the raw sensor data 210. For example, using a higher brightness ISP setting for a brightness ISP parameter may cause the sensor signal processor 220 applying the brightness ISP parameter to increase the brightness of the raw sensor data 210 more than a lower brightness ISP setting. Figure 2 , a graphic representing the sensor signal processor 220 illustrates a slider representing selection of an ISP setting from among different possible ISP settings for each ISP parameter, and a gear representing processing of the raw sensor data 210 according to the selected ISP setting for each ISP parameter.

[0051] The imaging system 200 includes a context controller engine 215. In some examples, the context controller engine receives raw sensor data 210 from the sensor 205. The context controller engine 215 identifies a context for generating sensor data (e.g., first sensor data 230 or second sensor data 235) based on the raw sensor data 210. The context controller engine 215 may also identify or set different ISP settings for different ISP parameters of the sensor signal processor 220 based on different contexts. For example, a first context may be associated with a first set of ISP settings for ISP parameters of the sensor signal processor 220, a second context may be associated with a second set of ISP settings for ISP parameters of the sensor signal processor 220, and so on. The different contexts may include a reason for generating the sensor data, how the sensor data will be output, how the sensor data will be used after being output, what the sensor data will be used for after being output, who will use the sensor data after being output, or a combination thereof.

[0052] exist Figure 2In the illustrated example, the first context includes generating first sensor data 230 (e.g., a first image) from raw sensor data 210 so that the first sensor data 230 can be displayed (in some cases, with some post-processing using a post-processing engine 240) to one or more human viewers of the first sensor data 230 using an output device 250 (e.g., a display screen). A first set of ISP settings associated with the first context can be optimized so that the first sensor data 230 appears clear, sharp, realistic, and / or color correct based on human vision of the one or more human viewers of the first sensor data 230.

[0053] exist Figure 2 In the illustrated example, the second context includes generating second sensor data 235 (e.g., a second image) from the raw sensor data 210, such that the second sensor data 235 can be further processed by a computer for use in a computer vision application (e.g., using one or more trained machine learning (ML) models 260 of a post-processing engine 245) and for use and / or output using an output device 255. In some examples, the post-processing engine 245 can process the second sensor data 235 using the trained ML model 260 to perform feature extraction, feature detection, feature recognition, feature tracking, object detection, object recognition, object tracking, facial detection, facial recognition, facial tracking, person detection, person recognition, person tracking, vehicle detection, vehicle recognition, vehicle tracking, classification, computer vision, or a combination thereof on the second sensor data 235. In some examples, output device 255 includes a vehicle and / or one or more subsystems of a vehicle, such as an advanced driver assistance system (ADAS) of a vehicle, an in-vehicle infotainment (IVI) system of a vehicle, an autonomous driving system of a vehicle, a semi-autonomous driving system of a vehicle, a vehicle electronic control unit (ECU) 630, or a combination thereof.

[0054] In some examples, the sensor signal processor 220 processes the raw sensor data 210 according to a first set of ISP settings associated with a first context (e.g., human vision) identified by the context controller engine 215 to the sensor signal processor 220 to generate first sensor data 230 (e.g., a first image). In some examples, the sensor signal processor 220 processes the raw sensor data 210 according to a second set of ISP settings associated with a second context (e.g., computer vision using a trained ML model 260 and / or a post-processing engine 245) identified by the context controller engine 215 to the sensor signal processor 220 to generate second sensor data 235 (e.g., a second image). The first set of ISP settings may be different from the second set of ISP settings. Therefore, the first sensor data 230 (e.g., the first image) may be different from the second sensor data 235 (e.g., the second image). For example, in some examples, human vision may be more sensitive to brightness than to color, while some computer vision systems are more equivalent in sensitivity to brightness and color. Therefore, the first set of ISP settings may adjust brightness and / or hue more than the second set of ISP settings. In some examples, human vision may be more sensitive to green than to other primary colors (e.g., red, green, and blue), while some computer vision systems are more equivalent in sensitivity to different primary colors. Therefore, the first set of ISP settings may adjust the green channel more than the second set of ISP settings. In some examples, for specific applications, some computer vision systems may not require color information at all, while human vision benefits from the colors in the image. Therefore, the first set of ISP settings may perform color-based filtering, while the second set of ISP settings may forgo performing color-based filtering, or may even convert the image to grayscale. In some examples, certain imaging operations are more important for human vision to enable humans to properly interpret images, such as white balance, tone mapping, color space transformation (STS), chromatic aberration correction (CAC) (e.g., longitudinal chromatic aberration correction (LCAC)), lens distortion correction, or a combination thereof. Therefore, the first set of ISP settings may perform stronger adjustments in these imaging operations than the second set of ISP settings, and in some examples, the second set of ISP settings may omit some of these imaging operations. In some examples, the second set of ISP settings may be tuned based at least in part on training and / or tuning of post-processing engine 245 and / or trained ML model 260 .

[0055] exist Figure 2, a graphic representing the context controller engine 215 illustrates a slash between a human eyeball icon and an ML model icon. The human eyeball icon represents a first context having ISP settings optimized for display to and viewing by a human viewer. The ML model icon represents a second context having ISP settings optimized for further processing and / or use by the ML model for a computing system (e.g., ADAS, IVI, autonomous driving, etc.). Figure 2 , the graphic representing first sensor data 230 illustrates a first image depicting the environment illustrated in the graphic representing sensor 205 using similar coloring as illustrated in the graphic representing sensor 205 to indicate that the first sensor data 230 is optimized for human vision. Figure 2 Within, the graphic representing second sensor data 235 illustrates a second image using significantly different coloring than the graphic representing sensor 205 to indicate that the second sensor data 235 is optimized for computer vision and may emphasize aspects of the raw sensor data 210 that are different from the first sensor data 230, the second image depicting the environment illustrated in the graphic representing sensor 205.

[0056] In some examples, the imaging system 200 may include a post-processing engine 240 associated with the first context, the post-processing engine for applying post-generation processing to the first sensor data 230 (e.g., the first image). The post-processing engine 240 may adjust one or more image attributes and / or other attributes of the first sensor data 230, such as brightness, contrast, color balance, tone mapping, sharpness, noise reduction, saturation, white balance, or a combination thereof. The post-processing engine 240 may output a modified variation of the first sensor data 230 to an output device 250. In some examples, the imaging system 200 may lack the post-processing engine 240, and the first sensor data 230 may be output directly to the output device 250. Figure 2 Within, a graphic representing post-processing engine 240 illustrates the first image depicted in the graphic representing first sensor data 230, wherein flashes in the corners represent new adjustments to image properties, and a human eye icon represents that the adjustment may modify first sensor data 230 to further optimize first sensor data 230 for display to and / or viewing by one or more human viewers.

[0057] The imaging system 200 may include a post-processing engine 245 associated with the second context. In some examples, the post-processing engine 245 applies post-generation processing to the second sensor data 235 (e.g., the second image), for example, to adjust one or more image attributes and / or other attributes of the second sensor data 235 (as discussed with respect to the post-processing engine 240), for example, to further optimize the second sensor data 235 for use in a computer vision application. In some examples, the post-processing engine 245 performs computer vision processing operations such as feature extraction, feature detection, feature recognition, feature tracking, object detection, object recognition, object tracking, face detection, face recognition, face tracking, person detection, person recognition, person tracking, vehicle detection, vehicle recognition, vehicle tracking, classification, computer vision, or a combination thereof on the second sensor data 235 (e.g., using the trained ML model 260). For example, the post-processing engine 245 may identify, detect, recognize, and / or track one or more features, objects, faces, persons, vehicles, etc. within the second sensor data 235. The post-processing engine 245 may output the second sensor data 235 with any modifications and / or additional elements (e.g., bounding boxes of identified features, objects, faces, people, vehicles, etc.) to an output device 255. In some examples, the output device 255 includes a vehicle and / or one or more subsystems of a vehicle, such as an advanced driver assistance system (ADAS) of the vehicle, an in-vehicle infotainment (IVI) system of the vehicle, an autonomous driving system of the vehicle, a semi-autonomous driving system of the vehicle, a vehicle electronic control unit (ECU) 630, or a combination thereof. Figure 2 Within, a graphic representing a post-processing engine 245 illustrates the second image depicted in the graphic representing the second sensor data 235, with a bounding box superimposed representing the detected location of a face in the second image, and an ML model icon representing a trained ML model 260 that can be used to detect a face and / or generate a bounding box to identify the location of a face in the second sensor data 235.

[0058] In some examples, the post-processing engine 245 inputs the second sensor data 235 into the trained ML model 260 to detect, identify, and / or track features, objects, faces, persons, vehicles, etc. in the second sensor data 235. The post-processing engine 245 and / or the trained ML model 260 may output locations and / or areas where those elements (e.g., features, objects, faces, persons, vehicles, etc.) are detected in the second sensor data 235, may output the identified identities of those elements, may output the tracked paths of those elements over time, and / or may output predicted paths of those elements based on the tracked paths of those elements. The trained machine learning (ML) model 260 may be trained (e.g., by the imaging system 200) for use by the post-processing engine 245 using training data that includes sensor data (e.g., images) generated using a second set of ISP settings associated with a second context (e.g., as in the second sensor data 235) and a determined location of an element, an area including the element, an identity of the element, a tracked path of the element, a predicted path of the element, or a combination thereof.

[0059] The imaging system 200 includes an output device 250 and an output device 255. The output devices 250 to 255 may include one or more visual output devices, such as a display or a connector therefor. The output devices 250 to 255 may include one or more audio output devices, such as a speaker, a headset, and / or a connector therefor. The output devices 250 to 255 may include one or more of the output device 1135 and / or the communication interface 1140 of the computing system 1100. In some examples, the imaging system 200 causes the display of the output device 250 to display the first sensor data 230 (in some cases with modifications and / or processing by the post-processing engine 240 applied). In some examples, the imaging system 200 causes the display of the output device 255 to display the second sensor data 235 (in some cases with modifications and / or processing by the post-processing engine 245 applied).

[0060] In some examples, the output devices 250 to 255 include one or more transceivers. The transceiver may include a wired transmitter, a receiver, a transceiver, or a combination thereof. The transceiver may include a wireless transmitter, a receiver, a transceiver, or a combination thereof. The transceiver may include one or more of the output device 1135 and / or the communication interface 1140 of the computing system 1100. In some examples, the imaging system 200 causes the transceiver to transmit the first sensor data 230 (in some cases, the modification and / or processing performed by the post-processing engine 240 is applied) to the recipient device. In some examples, the imaging system 200 causes the transceiver to transmit the second sensor data 235 (in some cases, the modification and / or processing performed by the post-processing engine 245 is applied) to the recipient device. In some examples, the recipient device may include the HMD 310, the mobile phone 410, the vehicle 510, the vehicle ECU 630, the computing system 1100, or a combination thereof. In some examples, the receiving device may include a display, and data transmitted from the transceivers of output devices 250 to 255 to the receiving device may cause the display of the receiving device to display first sensor data 230 (e.g., with processing performed by post-processing engine 240 applied) and / or second sensor data 235 (e.g., with processing performed by post-processing engine 245 applied).

[0061] In some examples, the display of the output devices 250 to 255 of the imaging system 200 is used as an optical "see-through" display that allows light from the real-world environment (scene) surrounding the imaging system 200 to traverse (e.g., pass through) the display of the output devices 250 to 255 to one or both eyes of the user. For example, the display of the output devices 250 to 255 may be at least partially transparent, translucent, light-transmissive, light-transmissive, or a combination thereof. In an illustrative example, the display of the output devices 250 to 255 includes a transparent, translucent, and / or light-transmissive lens and a projector. The display of the output devices 250 to 255 may include a projector that projects virtual content (e.g., the first sensor data 230, the second sensor data 235, the processing elements associated with the post-processing engine 240, and / or the processing elements associated with the post-processing engine 245) onto the lens. The lens may be, for example, a lens of a pair of glasses, a lens of goggles, a contact lens, a lens of a head-mounted display (HMD) device, or a combination thereof. Light from the real world environment passes through the lens and reaches one or both eyes of the user. The projector may project virtual content (e.g., first sensor data 230, second sensor data 235, processing elements associated with post-processing engine 240, and / or processing elements associated with post-processing engine 245) onto the lens so that the virtual content appears superimposed on the user's view of the environment from the perspective of one or both eyes of the user. In some examples, the projector may project the virtual content onto one or both retinas of one or both eyes of the user, rather than onto the lens, which may be referred to as a virtual retinal display (VRD), a retinal scanning display (RSD), or a retinal projector (RP) display.

[0062] In some examples, the display of the output device 250 to 255 of the imaging system 200 is a digital "transparent" display that allows the user of the imaging system 200 and / or the recipient device to view the environment view by displaying the environment view on the display of the output device 250 to 255. For example, based on sensor data (e.g., images, videos, depth images, point clouds, other depth data, or a combination thereof) captured by one or more environment-oriented sensors in the sensor 205 (e.g., the first sensor data 230, the second sensor data 235, the processing elements associated with the post-processing engine 240, and / or the processing elements associated with the post-processing engine 245), the environment view displayed on the digital transparent display can be a view of the real-world environment surrounding the imaging system 200. The environment view displayed on the digital transparent display can be a virtual environment (e.g., as in VR), which in some cases can include elements based on the real-world environment (e.g., the boundaries of a room). The environment view displayed on the digital transparent display can be an augmented environment based on the real-world environment (e.g., as in AR). The environment view displayed on the digital pass-through display may be a hybrid environment based on the real world environment (e.g., as in MR). The environment view displayed on the digital pass-through display may include virtual content superimposed on other content otherwise incorporated into the environment view (e.g., first sensor data 230, second sensor data 235, processing elements associated with post-processing engine 240, and / or processing elements associated with post-processing engine 245).

[0063] exist Figure 2 In the figure, the graphic representing the output device 250 illustrates a display, a speaker, and a wireless transceiver that output a first image, wherein the human eye icon indicates that the output device 250 is configured to output to a human viewer so that the human viewer can access the output of the output device 250. In the figure, the first image is illustrated in the graphic representing the first sensor data 230 and the post-processing engine 240. Figure 2 Within, a graphic representing an output device 255 illustrates a display, a speaker, a wireless transceiver, and a vehicle that output a second image, wherein the vehicle and ML model icons represent that the output device 255 is configured to apply computer vision analysis to the output and / or use the output to control a computing system, such as an ADAS of a vehicle, an IVI system of a vehicle, an autonomous driving system of a vehicle, a semi-autonomous driving system of a vehicle, a vehicle ECU 630, or a combination thereof, the second image being illustrated in a graphic representing second sensor data 235 and a post-processing engine 245.

[0064] The trained ML model 260 may include one or more neural networks (NNs) (e.g., neural network 900), one or more convolutional neural networks (CNNs), one or more trained time-delay neural networks (TDNNs), one or more deep networks, one or more autoencoders, one or more deep belief networks (DBNs), one or more recurrent neural networks (RNNs), one or more generative adversarial networks (GANs), one or more conditional generative adversarial networks (cGANs), one or more other types of neural networks, one or more trained support vector machines (SVMs), one or more trained random forests (RFs), one or more computer vision systems, one or more deep learning systems, one or more classifiers, one or more transformers, or a combination thereof. Figure 2 , a graphic representing a trained ML model 260 illustrates a group of circles connected to another group of circles. Each of these circles may represent a node (e.g., node 916), a neuron, a perceptron, a layer, a portion thereof, or a combination thereof. The circles are arranged in columns. The white circles in the leftmost column represent input layers (e.g., input layer 910). The white circles in the rightmost column represent output layers (e.g., output layer 914). The two columns of shaded circles between the leftmost column of white circles and the rightmost column of white circles each represent a hidden layer (e.g., hidden layers 912A to 912N).

[0065] In some examples, the imaging system 200 includes a feedback engine 265 of the imaging system 200. The feedback engine 265 may detect feedback received from a user interface of the imaging system 200. The feedback may include feedback on the output of the output devices 250 to 255 (e.g., the first sensor data 230, the second sensor data 235, the processing elements associated with the post-processing engine 240, and / or the processing elements associated with the post-processing engine 245). The feedback engine 265 may detect feedback received from another engine of the imaging system 200 about one engine of the imaging system 200, such as whether one engine decides to use data from another engine. For example, the feedback engine 265 may detect whether the output device 255 (e.g., a vehicle subsystem) decides to use data generated by the post-processing engine 245 (e.g., detection, recognition, and / or tracking of an object). The feedback received by the feedback engine 265 may be positive feedback or negative feedback. For example, if one engine of the imaging system 200 uses data from another engine of the imaging system 200, or if positive feedback is received from a user through a user interface, the feedback engine 265 may interpret this as positive feedback. If one engine of the imaging system 200 rejects data from another engine of the imaging system 200, or if negative feedback is received from a user through a user interface, the feedback engine 265 may interpret this as negative feedback. Positive feedback may also be based on attributes of sensor data from the sensor 205, such as the user smiling, laughing, nodding, saying a positive statement (e.g., "yes," "confirm," "ok," "next"), or otherwise reacting positively to the output of one of the engines described herein, or an indication of them. Negative feedback may also be based on attributes of sensor data from the sensor 205, such as the user frowning, crying, shaking his head (e.g., making a "no" gesture), saying a negative statement (e.g., "no," "reject," "bad," "not this"), or otherwise reacting negatively to the output of one of the engines described herein, or an indication of them.

[0066] In some examples, the feedback engine 265 provides feedback as training data to one or more ML systems (e.g., the post-processing engine 245) of the imaging system 200 to update one or more trained ML models 260 of the imaging system 200. For example, the feedback engine 265 may provide feedback as training data to the ML system and / or the trained ML model 260 to update the training of the post-processing engine 245, the trained ML model 260, or a combination thereof. Positive feedback may be used to strengthen and / or enhance weights associated with the output of the ML system and / or the trained ML model 260, and / or weaken or remove weights other than the weights associated with the output of the ML system and / or the trained ML model 260. Negative feedback may be used to weaken and / or remove weights associated with the output of the ML system and / or the trained ML model 260, and / or strengthen and / or enhance weights other than the weights associated with the output of the ML system and / or the trained ML model 260.

[0067] It should be understood that references herein to sensor 205 and other sensors described herein (as image sensors) should be understood to also include other types of sensors that can produce output in the form of images, such as depth sensors that produce depth images and / or point clouds that can be represented in the form of images and / or rendered images of 3D models (e.g., RADAR, LIDAR, SONAR, SODAR, ToF, structured light). It should be understood that references herein to image data and / or images produced by such sensors may include any sensor data that can be output in the form of images, such as depth images, point clouds that can be represented in the form of images, and / or rendered images of 3D models.

[0068] In some examples, certain elements of the imaging system 200 (e.g., context controller engine 215, sensor signal processor 220, post-processing engine 240, post-processing engine 245, output device 250, output device 255, trained ML model 260, feedback engine 265, or a combination thereof) include software elements, such as an instruction set corresponding to a program, which is executed on a processor (such as processor 1110 of computing system 1100, image processor 150, host processor 152, ISP 154, or a combination thereof). In some examples, one or more of these elements of the imaging system 200 may include one or more hardware elements, such as a dedicated processor (e.g., processor 1110 of computing system 1100, image processor 150, host processor 152, ISP 154, or a combination thereof). In some examples, one or more of these elements of the imaging system 200 may include a combination of one or more software elements and one or more hardware elements.

[0069] Figure 3A 300 is a perspective view illustrating a head mounted display (HMD) 310 used as part of the imaging system 200. The HMD 310 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. The HMD 310 may be an example of the imaging system 200. The HMD 310 includes a first camera 330A and a second camera 330B along the front of the HMD 310. The first camera 330A and the second camera 330B may be examples of the sensors 205 of the imaging systems 200 to 200B. The HMD 310 includes a third camera 330C and a fourth camera 330D that face the user's eyes when the user's eyes face the display 340. The third camera 330C and the fourth camera 330D may be examples of the sensors 205 of the imaging systems 200 to 200B. In some examples, the HMD 310 may have only a single camera with a single image sensor. In some examples, the HMD 310 may include one or more additional cameras in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D. In some examples, the HMD 310 may include one or more additional sensors in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D, which may also include other types of sensors 205 of the imaging system 200. In some examples, the first camera 330A, the second camera 330B, the third camera 330C, and / or the fourth camera 330D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0070] The HMD 310 may include one or more displays 340 visible to a user 320 (wearing the HMD 310 on the head of the user 320). The one or more displays 340 of the HMD 310 may be examples of one or more displays of the output devices 250 to 255 of the imaging systems 200 to 200B. In some examples, the HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the left eye of the user 320 and a right viewfinder for the right eye of the user 320. The left viewfinder may be oriented so that the left eye of the user 320 sees the left side of the display. The right viewfinder may be oriented so that the right eye of the user 320 sees the right side of the display. In some examples, the HMD 310 may include two displays 340, including a left display that displays content to the left eye of the user 320 and a right display that displays content to the right eye of the user 320. The one or more displays 340 of the HMD 310 may be digital "pass-through" displays or optical "see-through" displays.

[0071] The HMD 310 may include one or more earpieces 335, which may be used as speakers and / or headphones to output audio to one or both ears of a user of the HMD 310 and may be an example of output devices 250 to 255. Figure 3A and Figure 3B 310, but it should be understood that the HMD 310 may include two earpieces, one for each ear of the user (left ear and right ear). In some examples, the HMD 310 may also include one or more microphones (not shown). The one or more microphones may be examples of sensors 205 of imaging systems 200 to 200B. In some examples, the audio output by the HMD 310 to the user through the one or more earpieces 335 may include or be based on audio recorded using the one or more microphones.

[0072] Figure 3B is an example Figure 3A310 is a perspective view 350 of a head mounted display (HMD) being worn by a user 320. The user 320 wears the HMD 310 on the head of the user 320 above the eyes of the user 320. The HMD 310 may capture images through a first camera 330A and a second camera 330B. In some examples, the HMD 310 displays one or more output images toward the eyes of the user 320 using a display 340. In some examples, the output images may include the first sensor data 230, the second sensor data 235, the processing elements associated with the post-processing engine 240, and / or the processing elements associated with the post-processing engine 245. The output images may be based on the images captured by the first camera 330A and the second camera 330B (e.g., the raw sensor data 210, the first sensor data 230, the second sensor data 235), for example, with virtual content (e.g., the processing elements associated with the post-processing engine 240 and / or the processing elements associated with the post-processing engine 245) superimposed thereon. The output image may provide a stereoscopic view of the environment, in some cases superimposed with virtual content and / or with other modifications. For example, the HMD 310 may display a first display image to the right eye of the user 320, the first display image being based on an image captured by the first camera 330A. The HMD 310 may display a second display image to the left eye of the user 320, the second display image being based on an image captured by the second camera 330B. For example, the HMD 310 may provide superimposed virtual content in the display image superimposed on the image captured by the first camera 330A and the second camera 330B. The third camera 330C and the fourth camera 330D may capture images of the eyes before, during, and / or after the user views the display image displayed by the display 340. In this way, sensor data from the third camera 330C and / or the fourth camera 330D may capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content. The earpiece 335 of the HMD 310 is illustrated as being in the ear of the user 320. The HMD 310 may output audio to the user 320 through the earpiece 335 and / or through another earpiece (not shown) of the HMD 310 in the other ear of the user 320 (not shown).

[0073] Figure 4A 4 is a perspective view 400 illustrating a front surface of a mobile phone 410 that includes a front-facing camera and can be used as part of the imaging system 200. The mobile phone 410 can be an example of the imaging system 200. The mobile phone 410 can be, for example, a cellular phone, a satellite phone, a portable game console, a music player, a fitness tracking device, a wearable device, a wireless communication device, a laptop device, a mobile device, any other type of computing device or computing system discussed herein, or a combination thereof.

[0074] The front surface 420 of the mobile phone 410 includes a display 440. The front surface 420 of the mobile phone 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B can be examples of sensors 205 of the imaging systems 200 to 200B. The first camera 430A and the second camera 430B can face the user, including the user's eyes, while displaying content (e.g., first sensor data 230, second sensor data 235, processing elements associated with the post-processing engine 240, and / or processing elements associated with the post-processing engine 245) on the display 440. The display 440 can be an example of a display of the output device 250 to 255 of the imaging system 200 to 200B.

[0075] The first camera 430A and the second camera 430B are illustrated in a bezel around the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be positioned in a notch or cutout cut out of the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be under-display cameras positioned between the display 440 and the rest of the mobile phone 410 so that light passes through a portion of the display 440 before reaching the first camera 430A and the second camera 430B. The first camera 430A and the second camera 430B of the perspective view 400 are front-facing cameras. The first camera 430A and the second camera 430B face a direction perpendicular to the planar surface of the front surface 420 of the mobile phone 410. The first camera 430A and the second camera 430B may be two cameras of one or more cameras of the mobile phone 410. In some examples, the front surface 420 of the mobile phone 410 may have only a single camera.

[0076] In some examples, the display 440 of the mobile phone 410 displays one or more output images to a user using the mobile phone 410. In some examples, the output images may include the first sensor data 230, the second sensor data 235, the processing elements associated with the post-processing engine 240, and / or the processing elements associated with the post-processing engine 245. The output images may be based on images captured by the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D (e.g., the raw sensor data 210, the first sensor data 230, the second sensor data 235), for example, with virtual content (e.g., the post-processing engine 240 and / or the processing elements associated with the post-processing engine 245) superimposed thereon.

[0077] In some examples, the front surface 420 of the mobile phone 410 may include one or more additional cameras in addition to the first camera 430A and the second camera 430B. The one or more additional cameras may also be examples of sensors 205 of the imaging systems 200 to 200B. In some examples, the front surface 420 of the mobile phone 410 may include one or more additional sensors in addition to the first camera 430A and the second camera 430B. The one or more additional sensors may also be examples of sensors 205 of the imaging systems 200 to 200B. In some cases, the front surface 420 of the mobile phone 410 includes more than one display 440. The one or more displays 440 of the front surface 420 of the mobile phone 410 may be examples of displays of the output devices 250 to 255 of the imaging systems 200 to 200B. For example, the one or more displays 440 may include one or more touch screen displays.

[0078] The mobile handset 410 may include one or more speakers 435A and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4A 4. Although one speaker 435A is illustrated in the figure, it should be understood that the mobile phone 410 may include more than one speaker and / or other audio devices. In some examples, the mobile phone 410 may also include one or more microphones (not shown). The one or more microphones may be examples of sensors 205 of the imaging systems 200 to 200B. In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the front surface 420 of the mobile phone 410, where these microphones are examples of sensors 205 of the imaging systems 200 to 200B. In some examples, the audio output to the user by the mobile phone 410 through one or more speakers 435A and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0079] Figure 4B 4 is a perspective view 450 illustrating a rear surface 460 of a mobile phone that includes a rear camera and can be used as part of the imaging system 200. The mobile phone 410 includes a third camera 430C and a fourth camera 430D on the rear surface 460 of the mobile phone 410. The third camera 430C and the fourth camera 430D of the perspective view 450 are rear-facing. The third camera 430C and the fourth camera 430D can be Figure 2 The third camera 430C and the fourth camera 430D face a direction perpendicular to the planar surface of the rear surface 460 of the mobile phone 410.

[0080] The third camera 430C and the fourth camera 430D may be two cameras of the one or more cameras of the mobile phone 410. In some examples, the rear surface 460 of the mobile phone 410 may have only a single camera. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional cameras in addition to the third camera 430C and the fourth camera 430D. The one or more additional cameras may also be examples of the sensor 205 of the imaging system 200 to 200B. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional sensors in addition to the third camera 430C and the fourth camera 430D. The one or more additional sensors may also be examples of the sensor 205 of the imaging system 200 to 200B. In some examples, the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0081] The mobile handset 410 may include one or more speakers 435B and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4B 435B, but it should be understood that the mobile phone 410 may include more than one speaker and / or other audio devices. In some examples, the mobile phone 410 may also include one or more microphones (not shown). The one or more microphones may be examples of sensors 205 of the imaging systems 200 to 200B. In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the rear surface 460 of the mobile phone 410, where these microphones are examples of sensors 205 of the imaging systems 200 to 200B. In some examples, the audio output to the user by the mobile phone 410 through one or more speakers 435B and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0082] The mobile phone 410 can use the display 440 on the front surface 420 as a transparent display. For example, the display 440 can display an output image, such as the first sensor data 230, the second sensor data 235, the processing elements associated with the post-processing engine 240, and / or the processing elements associated with the post-processing engine 245. The output image can be based on the image captured by the third camera 430C and / or the fourth camera 430D (e.g., the raw sensor data 210, the first sensor data 230, and / or the second sensor data 235), for example, with virtual content (e.g., the processing elements associated with the post-processing engine 240 and / or the processing elements associated with the post-processing engine 245). The first camera 430A and / or the second camera 430B can capture an image of the user's eyes (and / or other parts of the user) before, during, and / or after the output image with virtual content is displayed on the display 440. In this way, sensor data from the first camera 430A and / or the second camera 430B may capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content.

[0083] Figure 5 500 is a perspective view illustrating a vehicle 510 including various sensors. The vehicle 510 can be an example of an imaging system 200. The vehicle 510 can be, for example, a car, a truck, a bus, a train, a ground vehicle, an airplane, a helicopter, an aircraft, an air vehicle, a ship, a submarine, a water vehicle, an underwater vehicle, a hovercraft, or a combination thereof. In some examples, the vehicle 510 can be manned. In some examples, the vehicle 510 can be unmanned, automatic and / or semi-automatic. In some examples, the vehicle can be controlled and / or used at least in part with a subsystem of the vehicle 510 (such as an ADAS of the vehicle 510, an IVI system of the vehicle 510, an automatic driving system of the vehicle 510, a semi-automatic driving system of the vehicle 510, a vehicle electronic control unit (ECU) 630 of the vehicle 510, or a combination thereof).

[0084] The vehicle 510 includes a display 520. The vehicle 510 includes various sensors, all of which may be examples of sensors 205. The vehicle 510 includes a first camera 530A and a second camera 530B located at the front, a third camera 530C and a fourth camera 530D located at the rear, and a fifth camera 530E and a sixth camera 530F located at the top. The vehicle 510 includes a first microphone 535A located at the front, a second microphone 535B located at the rear, and a third microphone 535C located at the top. The vehicle 510 includes a first sensor 540A located on one side (e.g., adjacent to one rearview mirror) and a second sensor 540B located on the other side (e.g., adjacent to another rearview mirror). The first sensor 540A and the second sensor 540B may include cameras, microphones, RADAR sensors, LIDAR sensors, or any other type of sensor 205 described herein. In some examples, in addition to Figure 5 In addition to the illustrated sensors, the vehicle 510 may include additional sensors 205. In some examples, the vehicle 510 may lack Figure 5 Some of the sensors illustrated.

[0085] In some examples, the display 520 of the vehicle 510 displays one or more output images to a user of the vehicle 510 (e.g., the driver and / or one or more passengers of the vehicle 510). In some examples, the output images may include the first sensor data 230, the second sensor data 235, the processing elements associated with the post-processing engine 240, and / or the processing elements associated with the post-processing engine 245. The output images may be based on images captured by the first camera 540A, the second camera 530B, the third camera 530C, the fourth camera 530D, the fifth camera 530E, the sixth camera 530F, the first sensor 530A, and / or the second sensor 540B (e.g., raw sensor data 210, first sensor data 230, second sensor data 235), for example, with virtual content (e.g., the post-processing engine 240 and / or the processing elements associated with the post-processing engine 245) superimposed thereon.

[0086] Figure 66 is a block diagram illustrating a row-interleaved multi-sensor multi-context imaging process performed using an imaging system 600. The imaging system 600 includes a plurality of sensors 605A to 605N. Each of the sensors 605A to 605N is an example of a sensor 205. In some examples, the sensors 605A to 605N are image sensors, such as image sensors 130. In some aspects, each of the sensors 605A to 605N may be asynchronous relative to at least one other sensor in the sensors 605A to 605N. In some aspects, all of the sensors in the sensors 605A to 605N may be asynchronous relative to each other. In some cases, at least two of the sensors 605A to 605N may be synchronized relative to each other. In some examples, the frame rate and / or resolution of each of the sensors 605A to 605N may be different from at least one other sensor in the plurality of sensors, or all of the sensors 605A to 605N may have different frame rates and / or resolutions.

[0087] The imaging system 600 includes Figure 2 Context controller engine 215 and sensor signal processor 220 are described. In some examples, sensor signal processor 220 is an ISP, such as ISP 154. Context controller engine 215 can select settings corresponding to different contexts. In some examples, first setting 610A corresponds to a first context configured for human vision. In some examples, second setting 610B corresponds to a first context configured for computer vision.

[0088] In some examples, the sensor signal processor 220 processes a portion (e.g., one row) of sensor data (e.g., image data) from the sensors 605A to 605N at a time. In some examples, the sensor signal processor 220 processes the sensor data (e.g., image data) from the sensors 605A to 605N in an interleaved manner such that the sensor signal processor 220 processes corresponding portions (e.g., corresponding rows) of sensor data from different sensors 605A to 605N that are interleaved between each other, and / or such that the sensor signal processor 220 processes corresponding portions (e.g., corresponding rows) of sensor data that are interleaved between each other with different settings (e.g., first settings 610A and second settings 610B). For example, Figure 6An example row interleaved order of processing operations performed by the sensor signal processor 220 is illustrated below the sensor signal processor 220 in FIG. The interleaved order of processing operations performed by the sensor signal processor 220 indicates that the sensor signal processor 220 processes row 0 of the sensor data from the sensor 605A according to the first setting 610A, then processes row 0 of the sensor data from the sensor 605A according to the second setting 610B, then processes row 0 of the sensor data from the sensor 605B according to the first setting 610A, then processes row 0 of the sensor data from the sensor 605B according to the second setting 610B, and then processes row 0 of the sensor data from the sensor 605C according to the first setting 610A. The process proceeds to row 0 of the sensor data from sensor 605A according to the first setting 610A, then row 1 of the sensor data from sensor 605A is processed according to the second setting 610B, and so on until row 0 of the sensor data from sensor 605N is processed according to the first setting 610A, then row 0 of the sensor data from sensor 605N is processed according to the second setting 610B, and then moving from row 0 to row 1 to process row 1 of the sensor data from sensor 605A according to the first setting 610A, then row 1 of the sensor data from sensor 605A is processed according to the second setting 610B, and so on.

[0089] The sensor signal processor 220 ultimately generates an output 640A from the sensor 605A processed according to the first setting 610A, which is output using an output device 620. The output device 620 is an example of the post-processing engine 240 and / or the output device 250. The sensor signal processor 220 generates an output 640B from the sensor 605A processed according to the second setting 610B, which is processed using the ML model processor 625 (e.g., the post-processing engine 245 and / or the trained ML model 260) and output to the vehicle ECU 630 of the vehicle 510 (e.g., an example of the output device 255).

[0090] Similarly, the sensor signal processor 220 generates an output 640C from the sensor 605B processed according to the first setting 610A, which is output using the output device 620. The sensor signal processor 220 generates an output 640D from the sensor 605B processed according to the second setting 610B, which is processed using the ML model processor 625 and output to the vehicle ECU 630 of the vehicle 510. The sensor signal processor 220 generates an output 640E from the sensor 605C processed according to the first setting 610A, which is output using the output device 620. The sensor signal processor 220 generates an output 640F from the sensor 605C processed according to the second setting 610B, which is processed using the ML model processor 625 and output to the vehicle ECU 630 of the vehicle 510. The sensor signal processor 220 generates an output 640G from the sensor 605N processed according to the first setting 610A, which is output using the output device 620. The sensor signal processor 220 generates an output 640H from the sensor 605N processed according to the second setting 610B, which is processed using the ML model processor 625 and output to the vehicle ECU 630 of the vehicle 510 .

[0091] In some examples, the context controller engine 215 can store and / or copy a portion (e.g., a row) of the sensor data and assign different contexts and, therefore, different ISP settings (e.g., first settings 610A and second settings 610B) to the corresponding copied portions. Processing the sensor data portion by portion allows the context controller engine 215 to store and / or copy less data at a time.

[0092] Figure 7 is a block diagram illustrating a frame interleaved multi-context imaging process performed using imaging system 700. Imaging system 700 includes sensor 705. Sensor 705 is an example of sensor 205. In some examples, sensor 705 is an image sensor, such as image sensor 130.

[0093] The imaging system 700 includes Figure 2 and Figure 6 The context controller engine 215 and the sensor signal processor 220 are described. In some examples, the sensor signal processor 220 is an ISP, such as ISP 154. The context controller engine 215 can select settings corresponding to different contexts, including, for example, Figure 6 The first setting 610A and the second setting 610B.

[0094] In some examples, the sensor signal processor 220 may have a high vertical blanking interval (VBI) requirement. The sensor 705 has a low vertical blanking interval (VBI), which means that it may take a long time to flush old sensor data from the sensor 705. To this end, the context controller engine 215 may copy a frame of data from the sensor 705 and process the frame according to each of the different ISP settings (e.g., the first setting 610A and the second setting 610B), thereby effectively treating the different copies of the sensor data as coming from Figure 6 By the time this processing is complete, old frame data is flushed from sensor 705, and new frame data may be captured by sensor 705 and processed using sensor signal processor 220 according to various ISP settings.

[0095] In some examples, the VBI of the sensor 705 may be indicated as or used as an indication of the size of data from a frame that may be flushed within a given time interval, such as the number of lines of frame data, the number of pixels of frame data, etc. For example, in the illustrative example, the VBI requirement of the sensor 705 may be 100 lines. Thus, in some examples, the VBI may specify the size of data from a frame to be processed by the sensor signal processor 220 using different ISP settings.

[0096] Figure 6 An example frame interleaved order of processing operations performed by the sensor signal processor 220 is illustrated below the sensor signal processor 220 in FIG. The interleaved order of processing operations performed by the sensor signal processor 220 indicates that the sensor signal processor 220 processes frame 0 of sensor data from the sensor 705 according to the first setting 610A, then processes frame 0 of sensor data from the sensor 705 according to the second setting 610B, then processes frame 1 of sensor data from the sensor 705 according to the first setting 610A, then processes row 1 of sensor data from the sensor 705 according to the second setting 610B, and so on.

[0097] The sensor signal processor 220 ultimately generates an output 740A from the sensor 705 processed according to the first setting 610A, which is output using the output device 620. The output device 620 is an example of the post-processing engine 240 and / or the output device 250. The sensor signal processor 220 generates an output 740B from the sensor 705 processed according to the second setting 610B, which is processed using the ML model processor 625 (e.g., the post-processing engine 245 and / or the trained ML model 260) and output to the vehicle ECU 630 of the vehicle 510 (e.g., an example of the output device 255).

[0098] Figure 8 800 is a block diagram illustrating an example architecture of a multi-context imaging process performed using an imaging system 800. The imaging system 800 may be part of and / or may include the context controller engine 215 and / or the sensor signal processor 220. The imaging system 800 includes an imaging engine 805, which may include software elements, hardware elements, or a combination thereof. In some cases, the imaging engine 805 may be a processor that is executed on a host computer (e.g., Figure 1 The imaging engine 805 may be software running on a host processor 152 of the ISP, wherein the software may configure settings for multiple contexts (e.g., context 825, context 830, and context 835) into the ISP hardware. In some aspects, the imaging engine 805 may be part of the context controller engine 215 and / or the sensor signal processor 220 and / or may include the context controller engine and / or the sensor signal processor.

[0099] The imaging engine 805 may deliver a context configuration 875 (e.g., human vision versus computer vision and / or corresponding ISP settings) to the demultiplexer 810. The imaging engine 805 includes three context data storage circuit elements, referred to as context 825, context 830, and context 835. Each of these context data storage circuit elements may store context configurations (e.g., including ISP settings) corresponding to different sensors, different contexts (e.g., human vision versus computer vision), or a combination thereof. The imaging engine 805 may use a write select register 820 to control which context configuration is stored in which of the three context data storage circuit elements. In some examples, the same context configuration may be used for all three context data storage circuit elements such as those that can be controlled by the imaging engine 805 via a broadcast write 815, for example, where these context data storage circuit elements correspond to different sensors. The broadcast write 815 allows the same context configuration to be used for all three context data storage circuit elements without using a write select register 820 to configure the three context data storage circuit elements individually.

[0100] Imaging system 800 includes a multiplexer 840 through which imaging engine 805 can read a context configuration stored in any of the context data storage circuit elements (context 825 , context 830 , and context 835 ) that can be controlled using a read select register 845 .

[0101] The imaging engine 805 includes a multi-context setup engine 855 that receives various context configurations as selected by the multi-context setup engine 855 using a multi-context setup selection register 860 from context data storage circuit elements (context 825, context 830, and context 835) via a multiplexer 850. The multi-context setup engine 855 may include software elements, hardware elements, or a combination thereof. The multi-context setup engine 855 may be part of the context controller engine 215 and / or the sensor signal processor 220, and / or may include the context controller engine and / or the sensor signal processor. In an example where the sensor signal processor 220 is set to process data from different sensors and / or different portions (e.g., different rows) according to different context configurations (e.g., different ISP settings), the multi-context setting engine 855 may determine which of these context configurations to extract from the context data storage circuit elements (context 825, context 830, and context 835) (e.g., via the multi-context setting selection register 860) based on which sensor's data is set to be processed next and which portion (e.g., which row) is set to be processed next.

[0102] In some examples, the context data storage circuit elements (context 825, context 830, and context 835) may each store two instances of the context configuration stored therein. In some examples, one instance is used for reading by the imaging engine 805 through the multiplexer 840, and the other instance is used for reading by the multi-context setup engine 855 through the multiplexer 850. This duplication may allow both the imaging engine 805 and the multi-context setup engine 855 to read the context configuration from the context data storage circuit elements (context 825, context 830, and context 835) at the same time.

[0103] The Parity 870 system can be stored in Figure 8 The parity check 870 system may provide parity checks for various elements of data in the imaging system 800, such as various instances of sensor data for various contexts. If one or more bits of data accidentally flip, such as due to a malfunction or a stray gamma ray hitting the circuitry of the imaging system 800, the parity check 870 system may trigger an alert. For applications where data integrity is critical, such as computer vision for vehicles, the parity check 870 system may be an important safety feature, where erroneous movement of the vehicle may result in an accident.

[0104] In some examples, an imaging system (e.g., imaging system 200, imaging system 600, imaging system 700, and / or imaging system 800) may include a data repository having a lookup table (LUT) and / or a switch (SWI) that may be used to store a correspondence between a specific context (e.g., human vision versus computer vision) and a corresponding ISP setting.

[0105] Fig. 9 is a block diagram illustrating an example of a neural network (NN) 900 that can be used for media processing operations. The neural network 900 may include any type of deep network, such as a convolutional neural network (CNN), an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), a generative adversarial network (GAN), and / or other types of neural networks. The neural network 900 may be an example of one of the post-processing engine 245, the trained ML model 260, the ML model processor 625, or a combination thereof. The neural network 900 may be used by the post-processing engine 245, the output device 255, the vehicle ECU 630, or a combination thereof.

[0106] The input layer 910 of the neural network 900 includes input data. The input data of the input layer 910 may include data representing pixels of one or more input image frames. In some examples, the input data of the input layer 910 includes data representing pixels of image data (e.g., an image captured by the image capture and processing system 100, raw image data of the raw sensor data 210, a first image of the first sensor data 230, a second image of the second sensor data 235, an image captured by one of the cameras 330A to 330D, an image captured by one of the cameras 430A to 430D, an image captured by one of the cameras 530A to 530F, an image captured by one of the sensors 605A to 605N, an image captured by the sensor 705, raw image data of operation 1005, or a combination thereof).

[0107] The image may include image data from an image sensor, including raw pixel data (including a single color per pixel based on, for example, a Bayer color filter) or processed pixel values ​​(e.g., RGB pixels of an RGB image). Neural network 900 includes a plurality of hidden layers 912, 912B through 912N. Hidden layers 912, 912B through 912N include "N" hidden layers, where "N" is an integer greater than or equal to 1. The plurality of hidden layers may include as many layers as required for a given application. Neural network 900 also includes an output layer 914 that provides output resulting from the processing performed by hidden layers 912, 912B through 912N.

[0108] In some examples, the output layer 914 may provide output images, such as the first sensor data 230, the second sensor data 235, the post-processed elements generated by the post-processing engine 240, the post-processed elements generated by the post-processing engine 245, the images output by the output device 250, the images output by the output device 255, the images output by the output device 620, the images generated using the ML model processor 625, the outputs 640A to 640H, the outputs 740A to 740B, or a combination thereof. In some examples, the output layer 914 may also provide other types of data, such as data (e.g., a bounding box of the post-processing engine 245) identifying the location, area, identity, tracked path, and / or predicted path of an element (e.g., a feature, object, face, person, vehicle, or a combination thereof) detected, recognized, and / or tracked using the NN 900.

[0109] Neural network 900 is a multi-layer neural network of interconnected filters. Each filter can be trained to learn features that represent input data. Information associated with these filters is shared between different layers, and each layer retains information as it processes it. In some cases, neural network 900 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back into itself. In some cases, network 900 may include a recurrent neural network, which may have loops that allow information to be carried across nodes as input is read in.

[0110] In some cases, information can be exchanged between layers by node-to-node interconnection between each layer. In some cases, the network may include a convolutional neural network, which may not link each node in a layer to each other node in the next layer. In a network in which information is exchanged between layers, the nodes of the input layer 910 may activate a set of nodes in the first hidden layer 912A. For example, as shown in the figure, each input node in the input nodes of the input layer 910 may be connected to each node in the nodes of the first hidden layer 912A. The nodes of the hidden layer may transform the information by applying an activation function (e.g., a filter) to the information of each input node. Then, the information derived from the transformation may be passed to the nodes of the next hidden layer 912B and may activate these nodes, which may perform their own specified functions. Example functions include convolution functions, reduction, enlargement, data transformation and / or any other suitable functions. Then, the output of the hidden layer 912B may activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 912N may activate one or more nodes of the output layer 914, which provides a processed output image. In some cases, although a node in neural network 900 (e.g., node 916) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0111] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 900. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have a tunable digital weight that can be tuned (e.g., based on a training data set), thereby allowing neural network 900 to adapt to the input and be able to learn as more and more data is processed.

[0112] The neural network 900 is pre-trained to process features of data from the input layer 910 using different hidden layers 912 , 912B to 912N to provide an output through an output layer 914 .

[0113] Fig.10 is a flow diagram illustrating a process 1000 for imaging. The process 1000 for imaging may be performed by an imaging system or a component of an imaging system (eg, a chipset, a processor or processors (such as an ISP, HP or other processor), or other component). In some examples, the imaging system may include, for example, image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, imaging system 200, context controller engine 215, sensor signal processor 220, post-processing engine 240, post-processing engine 245, output device 250, output device 255, trained ML model 260, feedback engine 265, HMD 310, mobile phone 410, vehicle 510, imaging system 600, sensors 605A to 605N, output device 620, ML model processor 625, vehicle ECU 630, imaging system 700, imaging system 800, neural network 900, computing system 1100, processor 1110, or a combination thereof. In some examples, the imaging system includes a display. In some examples, the imaging system includes a transceiver.

[0114] At operation 1005, the imaging system (or a component thereof) may receive raw image data captured using an image sensor. Illustrative examples of image sensors include image sensor 130, sensor 205, first camera 330A, second camera 330B, third camera 330C, fourth camera 330D, first camera 430A, second camera 430B, third camera 430C, fourth camera 430D, first camera 530A, second camera 530B, third camera 530C, fourth camera 530D, fifth camera 530E, sixth camera 530F, first sensor 540A, second sensor 540B, sensors 605A-605N, sensor 705, an image sensor for capturing an image used as input data for input layer 910 of NN 900, input device 1145, another image sensor described herein, another sensor described herein, or a combination thereof. Examples of this raw image data include raw image data 210.

[0115] At operation 1010, the imaging system (or a component thereof) may process the raw image data according to a first ISP setting (e.g., using an ISP) to generate a first image. At operation 1015, the imaging system (or a component thereof) may process the raw image data according to a second ISP setting (e.g., using an ISP) to generate a second image. The second ISP setting is different from the first ISP setting. For example, according to some aspects, the second ISP setting is different from the first ISP setting based on at least one difference in at least one of brightness, color balance, tone mapping, sharpness, noise reduction, saturation, or white balance.

[0116] In some aspects, the first ISP setting and the second ISP setting are based on different contexts. In an illustrative example, the first ISP setting is associated with processing the raw image data for output, and the second ISP setting is associated with processing the raw image data for further processing using a trained machine learning model. For example, the first ISP setting associated with processing the raw image data for output may correspond to outputting the raw image data for display. Additionally or alternatively, in another example, the first ISP setting associated with processing the raw image data for output may correspond to outputting the raw image data for sending to a device. In some cases, the trained machine learning model is configured to perform the further processing for use in controlling a device or system, such as controlling the vehicle using an electronic control unit associated with a vehicle.

[0117] At operation 1020, the imaging system (or a component thereof) may output the first image and the second image. For example, the imaging system (or a component thereof) may output the first image and the second image by displaying the first image and / or the second image, by sending the first image and / or the second image to another device, a combination thereof, and / or outputting the first image and the second image in other ways.

[0118] In some aspects, the imaging system (or a component thereof) may process multiple portions of the raw image data sequentially to process the raw image data. The imaging system (or a component thereof) may interleave processing a portion of the raw image data according to the first ISP setting with processing the portion of the raw image data according to the second ISP setting. Illustrative examples of such interleaved processing are described in Figure 7 . In some cases, the multiple portions of the raw image data are multiple rows of the raw image data. In some cases, the multiple portions of the raw image data are multiple frames of the raw image data. In some aspects, the imaging system (or a component thereof) may determine the size of the multiple portions of the raw image data based on a vertical blanking interval (VBI).

[0119] In some cases, an imaging system (or a component thereof) may receive secondary raw image data captured using a secondary image sensor. The imaging system (or a component thereof) may process the secondary raw image data according to a third ISP setting to generate a third image. The imaging system (or a component thereof) may also process the secondary raw image data according to a fourth ISP setting to generate a fourth image. The imaging system (or a component thereof) may output the third image and the fourth image. In some examples, the third ISP setting is the same as the first ISP setting, and wherein the fourth ISP setting is the same as the second ISP setting. In some aspects, the imaging system (or a component thereof) may interleave processing of portions of the raw image data with processing of corresponding portions of the secondary raw image data.

[0120] In some aspects, the imaging system (or a component thereof) may receive additional raw image data captured using an image sensor. The imaging system (or a component thereof) may process the additional raw image data according to the second ISP setting to generate a third image, wherein the first ISP setting is not used to generate the third image. The imaging system (or a component thereof) may also output the first image and the second image.

[0121] In some examples, the processes described herein (e.g., Figure 1 , Figure 2 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 The corresponding process Fig.10In some examples, the processes described herein may be performed by the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the imaging system 200, the context controller engine 215, the sensor signal processor 220, the post-processing engine 240, the post-processing engine 245, the output device 250, the output device 255, the trained ML model 260, the feedback engine 265, the HMD 310, the mobile phone 410, the vehicle 510, the imaging system 600, the sensors 605A to 605N, the output device 620, the ML model processor 625, the vehicle ECU 630, the imaging system 700, the imaging system 800, the neural network 900, the imaging system performing the process 1000, the computing system 1100, the processor 1110, or a combination thereof.

[0122] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR head-mounted device, an AR head-mounted device, an AR glasses, a network-connected watch or smart watch or other wearable device), a server computer, an automatic vehicle or a computing device of an automatic vehicle, a robotic device, a television, and / or any other computing device with resource capabilities to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0123] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0124] The processes described herein are illustrated as logic flow diagrams, block diagrams, or conceptual diagrams, the operations of which represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0125] Additionally, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination of the above. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0126] Fig.11 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.11 An example of a computing system 1100 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1105. The connection 1105 may be a physical connection using a bus, or a direct connection into the processor 1110, such as in a chipset architecture. The connection 1105 may also be a virtual connection, a networked connection, or a logical connection.

[0127] In some aspects, computing system 1100 is a distributed system, where the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, i.e., each of these components performs some or all of the functionality for which the component is described. In some aspects, each component may be a physical or virtual device.

[0128] The example system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 that couple various system components including system memory 1115, such as read only memory (ROM) 1120 and random access memory (RAM) 1125, to the processor 1110. The computing system 1100 may include a cache 1112 of high-speed memory directly connected to the processor 1110, in close proximity to the processor, or integrated as part of the processor.

[0129] Processor 1110 may include any general purpose processor and hardware or software services, such as services 1132, 1134, and 1136 stored in storage device 1130, that are configured to control processor 1110 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1110 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0130] To enable user interaction, the computing system 1100 includes an input device 1145 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1100 may also include an output device 1135 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, Low energy (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 1102.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1140 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 1100 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0131] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0132] Storage device 1130 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1110, causes the system to perform functions. In some aspects, hardware services that perform a particular function may include software components for performing functions stored in a computer-readable medium connected to the necessary hardware components (such as processor 1110, connection 1105, output device 1135, etc.).

[0133] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0134] In some aspects, computer readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0135] Specific details are provided in the above description to provide a detailed understanding of the various aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in software or a combination of hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0136] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0137] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0138] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0139] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0140] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that, in alternative aspects, each method can be performed in a different order than described.

[0141] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0142] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0143] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0144] Claim language or other language that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that states "at least one of A and B" means A, B, or A and B. In another example, claim language that states "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that states "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0145] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0146] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions, which execute one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0147] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0148] Illustrative aspects of the present disclosure include:

[0149] Aspect 1. A device for user role management, the device comprising: a memory; and at least one processor (for example, implemented in a circuit), the at least one processor coupled to the memory and configured to: receive raw image data captured using an image sensor; process the raw image data according to a first image signal processor (ISP) setting to generate a first image; process the raw image data according to a second ISP setting to generate a second image, wherein the second ISP setting is different from the first ISP setting; and output the first image and the second image.

[0150] Aspect 2. The apparatus according to aspect 1, wherein the first ISP setting and the second ISP setting are based on different contexts.

[0151] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein the first ISP setting is associated with processing the raw image data for output, and wherein the second ISP setting is associated with processing the raw image data for further processing using a trained machine learning model.

[0152] Clause 4. The apparatus of clause 3, wherein the first ISP setting associated with processing the raw image data for output corresponds to outputting the raw image data for display.

[0153] Clause 5. An apparatus according to any one of Clauses 3 or 4, wherein the first ISP setting associated with processing the raw image data for output corresponds to outputting the raw image data for transmission to a device.

[0154] Aspect 6. An apparatus according to any one of Aspects 3 to 5, wherein the trained machine learning model is configured to perform the further processing for use when controlling the vehicle using an electronic control unit associated with the vehicle.

[0155] Aspect 7. An apparatus according to any one of Aspects 1 to 6, wherein the at least one processor is configured to: process multiple portions of the original image data in sequence to process the original image data; and interleave a portion of the original image data processed according to the first ISP setting with a portion of the original image data processed according to the second ISP setting.

[0156] Clause 8. The apparatus according to clause 7, wherein the plurality of portions of the original image data are a plurality of lines of the original image data.

[0157] Clause 9. The apparatus according to clause 7, wherein the plurality of portions of the original image data are a plurality of frames of the original image data.

[0158] Aspect 10. An apparatus according to any one of aspects 7 to 9, wherein the at least one processor is configured to: determine the sizes of the multiple portions of the original image data based on a vertical blanking interval (VBI).

[0159] Aspect 11. An apparatus according to any one of Aspects 1 to 10, wherein the at least one processor is configured to: receive secondary raw image data captured using a secondary image sensor; process the secondary raw image data according to a third ISP setting to generate a third image; process the secondary raw image data according to a fourth ISP setting to generate a fourth image; and output the third image and the fourth image.

[0160] Aspect 12. The apparatus according to aspect 11, wherein the third ISP setting is the same as the first ISP setting, and wherein the fourth ISP setting is the same as the second ISP setting.

[0161] Aspect 13. An apparatus according to any one of Aspects 11 or 12, wherein the at least one processor is configured to interleave processing of a portion of the raw image data with processing of a corresponding portion of the secondary raw image data.

[0162] Aspect 14. An apparatus according to any one of Aspects 1 to 13, wherein the second ISP setting is different from the first ISP setting based on at least one difference in at least one of brightness, color balance, tone mapping, sharpness, noise reduction, saturation, or white balance.

[0163] Aspect 15. An apparatus according to any one of Aspects 1 to 14, wherein the at least one processor is configured to: receive additional raw image data captured using the image sensor; process the additional raw image data according to the second ISP setting to generate a third image, wherein the first ISP setting is not used to generate the third image; and output the first image and the second image.

[0164] Aspect 16. An apparatus according to any one of Aspects 1 to 15, wherein the at least one processor comprises an ISP.

[0165] Aspect 17. A method for multi-context image capture, the method comprising: receiving raw image data captured using an image sensor; using an image signal processor (ISP) to process the raw image data according to a first ISP setting to generate a first image; using the ISP to process the raw image data according to a second ISP setting to generate a second image, wherein the second ISP setting is different from the first ISP setting; and outputting the first image and the second image.

[0166] Aspect 18. The method according to aspect 17, wherein the first ISP setting and the second ISP setting are based on different contexts.

[0167] Aspect 19. A method according to any one of Aspects 17 to 18, wherein the first ISP setting is associated with processing the raw image data for output, and wherein the second ISP setting is associated with processing the raw image data for further processing using a trained machine learning model.

[0168] Clause 20. The method of clause 19, wherein the first ISP setting associated with processing the raw image data for output corresponds to at least one of outputting the raw image data for display.

[0169] Clause 21. The method of any one of clauses 19 or 20, wherein the first ISP setting associated with processing the raw image data for output corresponds to outputting the raw image data for transmission to a device.

[0170] Aspect 22. A method according to any one of Aspects 19 to 21, wherein the trained machine learning model is configured to perform the further processing for use when controlling the vehicle using an electronic control unit associated with the vehicle.

[0171] Aspect 23. A method according to any one of Aspects 17 to 22, the method further comprising: processing multiple portions of the original image data in sequence to process the original image data; and interleaving a portion of the original image data processed according to the first ISP setting with the portion of the original image data processed according to the second ISP setting.

[0172] Clause 24. The method according to clause 23, wherein the plurality of portions of the original image data are a plurality of lines of the original image data.

[0173] Clause 25. The method according to clause 23, wherein the plurality of portions of the original image data are a plurality of frames of the original image data.

[0174] Aspect 26. The method according to any one of aspects 23 to 25, further comprising: determining the sizes of the multiple portions of the original image data based on a vertical blanking interval (VBI).

[0175] Aspect 27. A method according to any one of Aspects 17 to 26, the method further comprising: receiving secondary raw image data captured using a secondary image sensor; processing the secondary raw image data according to a third ISP setting to generate a third image; processing the secondary raw image data according to a fourth ISP setting to generate a fourth image; and outputting the third image and the fourth image.

[0176] Aspect 28. The method according to aspect 27, wherein the third ISP setting is the same as the first ISP setting, and wherein the fourth ISP setting is the same as the second ISP setting.

[0177] Aspect 29. The method according to any one of aspects 27 or 28, further comprising: interleaving the processing of a portion of the original image data with a corresponding portion of the processing of the secondary original image data.

[0178] Aspect 30. A method according to any one of aspects 17 to 29, wherein the second ISP setting is different from the first ISP setting based on at least one difference in at least one of brightness, color balance, tone mapping, sharpness, noise reduction, saturation, or white balance.

[0179] Aspect 31. A method according to any one of Aspects 17 to 30, the method further comprising: receiving additional raw image data captured using the image sensor; using the ISP, processing the additional raw image data according to the second ISP setting to generate a third image, wherein the first ISP setting is not used to generate the third image; and outputting the first image and the second image.

[0180] Aspect 32. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of Aspects 1 to 31.

[0181] Aspect 33. An apparatus for image processing, the apparatus comprising one or more components for performing the operations according to any one of aspects 1 to 31.

Claims

1. A device for multi-context image capture, the device comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: receiving raw image data captured using an image sensor; processing the raw image data according to a first image signal processor (ISP) setting to generate a first image; processing the raw image data according to a second ISP setting to generate a second image, wherein the second ISP setting is different from the first ISP setting; as well as The first image and the second image are output. 2 . The apparatus of claim 1 , wherein the first ISP setting and the second ISP setting are based on different contexts.

3. The apparatus of claim 1 , wherein the first ISP setting is associated with processing the raw image data for output, wherein the second ISP setting is associated with processing the raw image data for further processing using a trained machine learning model. 4 . The apparatus of claim 3 , wherein the first ISP setting associated with processing the raw image data for output corresponds to outputting the raw image data for display.

5. The apparatus of claim 3, wherein the first ISP setting associated with processing the raw image data for output corresponds to outputting the raw image data for transmission to a device.

6. The apparatus of claim 3, wherein the trained machine learning model is configured to perform the further processing for use in controlling the vehicle using an electronic control unit associated with the vehicle.

7. The apparatus of claim 1 , wherein the at least one processor is configured to: sequentially processing the portions of the raw image data to process the raw image data; and Processing a portion of the raw image data according to the first ISP setting is interleaved with processing the portion of the raw image data according to the second ISP setting.

8. The apparatus of claim 7, wherein the plurality of portions of the raw image data are a plurality of lines of the raw image data.

9. The apparatus of claim 7, wherein the plurality of portions of the raw image data are a plurality of frames of the raw image data.

10. The apparatus of claim 7, wherein the at least one processor is configured to: The sizes of the portions of the raw image data are determined based on a vertical blanking interval (VBI).

11. The apparatus of claim 1 , wherein the at least one processor is configured to: receiving secondary raw image data captured using a secondary image sensor; processing the secondary raw image data according to a third ISP setting to generate a third image; processing the secondary raw image data according to a fourth ISP setting to generate a fourth image; as well as The third image and the fourth image are output.

12. The apparatus of claim 11, wherein the third ISP setting is the same as the first ISP setting, and wherein the fourth ISP setting is the same as the second ISP setting.

13. The apparatus of claim 11, wherein the at least one processor is configured to interleave processing of portions of the raw image data with processing of corresponding portions of the secondary raw image data.

14. The device of claim 1, wherein the second ISP setting is different from the first ISP setting based on at least one difference in at least one of brightness, color balance, tone mapping, sharpness, noise reduction, saturation, or white balance.

15. The apparatus of claim 1, wherein the at least one processor is configured to: receiving additional raw image data captured using the image sensor; processing the additional raw image data according to the second ISP settings to generate a third image, wherein the first ISP settings are not used to generate the third image; and The first image and the second image are output.

16. The apparatus of claim 1, wherein the at least one processor comprises an ISP.

17. A method for multi-context image capture, the method comprising: receiving raw image data captured using an image sensor; processing the raw image data using an image signal processor (ISP) according to a first ISP setting to generate a first image; processing the raw image data according to second ISP settings to generate a second image using the ISP, wherein the second ISP settings are different from the first ISP settings; as well as The first image and the second image are output.

18. The method of claim 17, wherein the first ISP setting and the second ISP setting are based on different contexts.

19. The method of claim 17, wherein the first ISP setting is associated with processing the raw image data for output, and wherein the second ISP setting is associated with processing the raw image data for further processing using a trained machine learning model.

20. The method of claim 19, wherein the first ISP setting associated with processing the raw image data for output corresponds to at least one of outputting the raw image data for display or outputting the raw image data for transmission to a device.

21. The method of claim 19, wherein the trained machine learning model is configured to perform the further processing for use in controlling the vehicle using an electronic control unit associated with the vehicle.

22. The method according to claim 17, further comprising: sequentially processing the plurality of portions of the raw image data to process the raw image data; as well as Processing a portion of the raw image data according to the first ISP setting is interleaved with processing the portion of the raw image data according to the second ISP setting.

23. The method of claim 22, wherein the plurality of portions of the raw image data are a plurality of lines of the raw image data.

24. The method of claim 22, wherein the plurality of portions of the raw image data are a plurality of frames of the raw image data.

25. The method according to claim 22, further comprising: The sizes of the portions of the raw image data are determined based on a vertical blanking interval (VBI).

26. The method of claim 17, further comprising: receiving secondary raw image data captured using a secondary image sensor; processing the secondary raw image data according to a third ISP setting to generate a third image; processing the secondary raw image data according to a fourth ISP setting to generate a fourth image; as well as The third image and the fourth image are output.

27. The method of claim 26, wherein the third ISP setting is the same as the first ISP setting, and wherein the fourth ISP setting is the same as the second ISP setting.

28. The method according to claim 26, further comprising: Processing portions of the raw image data is interleaved with processing corresponding portions of the secondary raw image data.

29. The method of claim 17, wherein the second ISP setting differs from the first ISP setting based on at least one difference in at least one of brightness, color balance, tone mapping, sharpness, noise reduction, saturation, or white balance.

30. The method of claim 17, further comprising: receiving additional raw image data captured using the image sensor; processing the additional raw image data according to the second ISP settings to generate a third image using the ISP, wherein the first ISP settings are not used to generate the third image; as well as The first image and the second image are output.

Images