Sensor interrupt handling

Abstract

Techniques for interrupt handling are described. The device can obtain first processing configuration data associated with the first sensor at a first context buffer associated with the first interrupt handler; obtaining second processing configuration data associated with a second sensor at a second context buffer associated with a second interrupt handler; obtaining, at the first interrupt handler, an error indication associated with the first sensor; performing, by the first interrupt handler, an error handling operation based on the error indication, wherein the error handling operation includes refreshing the first context buffer and / or invalidating first processing configuration data in the first context buffer; obtaining, at the second interrupt handler, an interrupt request (IRQ) from a second sensor during a time window interval between obtaining the error indication and a completion time of the error handling operation; and outputting the second processing configuration data to the processor based on the IRQ.

Description

TECHNICAL FIELD

[0001] Aspects of the disclosure relate to systems and techniques for performing sensor interrupt handling. BACKGROUND

[0002] Many devices and systems allow for capturing a scene by generating images (or frames) and / or video data (including multiple frames) of the scene. For example, a camera or a device including a camera can be able to capture a sequence of frames of a scene (e.g., a video of the scene). In some cases, the sequence of frames can be processed to perform one or more functions, can be output for display, can be output for processing and / or consumption by other devices, among other uses.

[0003] A vehicle is an example of a device that can include multiple cameras. For example, a vehicle can include one or more cameras that can capture images of the exterior of the vehicle. The images can be processed for various purposes, such as identifying other vehicles, objects, and / or obstacles in the vicinity of the vehicle, among others. SUMMARY

[0004] In some examples, systems and techniques for interrupt handling are described. According to at least one illustrative example, a method of interrupt handling is provided. The method includes obtaining, at a first context buffer associated with a first interrupt handler, first image processing configuration data associated with a first image sensor; obtaining, at a second context buffer associated with a second interrupt handler, second image processing configuration data associated with a second image sensor different from the first image sensor; obtaining, at the first interrupt handler, an error indication associated with the first image sensor; performing, by the first interrupt handler, an error handling operation based on the error indication associated with the first image sensor, wherein the error handling operation includes at least one of flushing the first context buffer or invalidating the first image processing configuration data in the first context buffer; obtaining, at the second interrupt handler, an interrupt request (IRQ) from the second image sensor during a time window interval between obtaining the error indication associated with the first image sensor and a completion time of the error handling operation; and outputting, based on obtaining the IRQ, the second image processing configuration data to an image processor.

[0005] In another illustrative example, an apparatus for interrupt handling includes: a first interrupt handler configured to obtain an error indication associated with a first image sensor; a first context buffer associated with the first interrupt handler and configured to obtain first image processing configuration data associated with the first image sensor; a second interrupt handler; and a second context buffer associated with the second interrupt handler and configured to obtain second image processing configuration data associated with a second image sensor different from the first image sensor; wherein the first interrupt handler is further configured to perform an error handling operation based on the error indication associated with the first image sensor, wherein the error handling operation includes at least one of refreshing the first context buffer or invalidating the first image processing configuration data in the first context buffer; and wherein the second interrupt handler is configured to obtain an interrupt request IRQ from the second image sensor during a time window interval between obtaining the error indication associated with the first image sensor and the completion time of the error handling operation; and wherein the second context buffer is configured to output the second image processing configuration data to an image processor based on the obtained IRQ.

[0006] In some aspects, one or more of the devices described above are, are part of, or include: vehicles (e.g., vehicle computing systems of vehicles), mobile devices (e.g., mobile phones or so-called "smartphones" or other mobile devices), wearable devices, augmented reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), personal computers, laptop computers, server computers, or other devices. In some aspects, a device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device can include one or more sensors. In some cases, one or more sensors can be used to determine the position and / or orientation of the device, the state of the device, and / or for other purposes.

[0007] This disclosure is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of the entire specification of this patent, any or all of the accompanying drawings, and each claim.

[0008] The foregoing, as well as other features and embodiments, will become more apparent from the following description, claims, and drawings. Attached Figure Description

[0009] The illustrative embodiments of this application are described in detail below with reference to the accompanying drawings:

[0010] Figure 1A and Figure 1A This is a block diagram illustrating an example multi-sensor camera configuration according to some examples of this disclosure;

[0011] Figure 2 This is a block diagram illustrating an example of a vehicle computing system according to some examples of this disclosure;

[0012] Figure 3 This is a block diagram illustrating an example of an additional multi-sensor camera configuration based on some examples of this disclosure;

[0013] Figure 4 This is a block diagram of a multi-sensor camera configuration including multiple interrupt handlers, based on some examples of this disclosure;

[0014] Figure 5 This is a flowchart illustrating an example of a process for processing one or more frames according to some examples of this disclosure;

[0015] Figure 6 This is a block diagram illustrating examples of deep learning networks according to some examples of this disclosure;

[0016] Figure 7 This is a block diagram illustrating examples of convolutional neural networks according to some examples of this disclosure;

[0017] Figure 8 This is a diagram illustrating examples of computing systems for implementing certain aspects described herein, according to some examples of this disclosure. Detailed Implementation

[0018] Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some can be applied in combination, as will be apparent to those skilled in the art. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of this application. However, it will be apparent, however, that various embodiments can be practiced without these specific details. The accompanying drawings and description are not intended to be limiting.

[0019] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of this application as set forth in the appended claims.

[0020] Systems and devices (e.g., autonomous vehicles, such as autonomous and semi-autonomous cars, drones, mobile robots, mobile devices, extended reality (XR) devices, and other suitable systems or devices) increasingly include multiple sensors for collecting information about the environment, and processing systems for processing the collected information, such as for route planning, navigation, collision avoidance, etc. An example of such a system is an Advanced Driver Assistance System (ADAS) for vehicles. Sensor data (such as images captured from one or more cameras) can be processed and / or analyzed to perform one or more functions, such as detecting objects (e.g., targets).

[0021] In some cases, a processing system (e.g., vehicle computing system 250) can be used to configure sensors, coordinate sensor data acquisition, and / or coordinate sensor data processing. In some implementations, multiple sensors can generate interrupt requests (IRQs) as signaling to communicate with the processing system. In some cases, an interrupt handler can coordinate sensor data processing. For example, an interrupt handler can coordinate sensor data processing based on IRQs generated by multiple sensors.

[0022] Figure 1A An example multi-sensor camera configuration 100 including a software (SW) interrupt handler is shown. Figure 1A As shown, the multi-sensor camera configuration 100 includes multiple sensors 102, multiple decoders 104, a processor 120, a programming engine 130, and an image processor 140.

[0023] In some cases, sensor 102 may include one or more image sensors. In some examples, image sensor 102 may include one or more arrays of photodiodes or other photosensitive elements. Figure 1A In the example, each image sensor 102 outputs data to a corresponding decoder 104. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by the image sensor 102. In some cases, different photodiodes can be covered by different filters. In some cases, different photodiodes can be covered in color filters, and thus light matching the color of the color filter covering the photodiodes can be measured. Various color filter arrays can be used, including Bayer color filter arrays, quaternary color filter arrays (also known as quaternary Bayer color filter arrays or QCFAs), and / or any other color filter array. For example, a Bayer color filter includes a red color filter, a blue color filter, and a green color filter, wherein each pixel of the image is generated based on red light data from at least one photodiode covered in the red color filter, blue light data from at least one photodiode covered in the blue color filter, and green light data from at least one photodiode covered in the green color filter.

[0024] In some examples, instead of red, blue, and / or green filters, or other types of filters besides red, blue, and / or green filters, yellow, magenta, and / or cyan (also known as "emerald green") filters can be used. In some cases, some photodiodes can be configured to measure infrared (IR) light. In some implementations, the photodiode measuring IR light may not be covered by any filters, thereby allowing the IR photodiode to measure both visible light (e.g., color) and IR light. In some examples, the IR photodiode may be covered by an IR filter, thereby allowing IR light to pass through and blocking light from other parts of the spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 102) may be completely devoid of filters (e.g., color, IR, or any other part of the spectrum) and may alternatively use different photodiodes throughout the pixel array (in some cases, vertically stacked). Different photodiodes throughout the pixel array can have different spectral sensitivity profiles and thus respond to light of different wavelengths. Monochrome image sensors may also lack filters, thus lacking color depth.

[0025] In some cases, the image sensor 102 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions thereof at certain times and / or from certain angles. In some cases, the opaque and / or reflective mask may be used for phase detection autofocus (PDAF). In some cases, the opaque and / or reflective mask may be used to block portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., IR cutoff filters, UV cutoff filters, bandpass filters, low-pass filters, high-pass filters, etc.). The image sensor 102 may also include an analog gain amplifier to amplify the analog signal output from the photodiodes and / or an analog-to-digital converter (ADC) to convert the analog signal output from the photodiodes (and / or amplified by the analog gain amplifier) ​​into a digital signal. Image sensor 102 may include charge-coupled device (CCD) sensor, electron multiplication CCD (EMCCD) sensor, active pixel sensor (APS), complementary metal-oxide semiconductor (CMOS), N-type metal-oxide semiconductor (NMOS), hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0026] exist Figure 1AIn the example shown, each image sensor 102 is coupled to a decoder 104. In some cases, the decoder can be used to decode serialized and / or encoded data streams from each image sensor 102. In one illustrative example, the image sensor 102 and decoder 104 can communicate via a Mobile Industry Processor Interface (MIPI) port or interface (e.g., a MIPI CSI-2 physical (PHY) layer port or interface). In some cases, the decoder can generate an interrupt request (IRQ) 105 indicating the completion of a sensor data acquisition operation (e.g., the end of an image frame). In some cases, decoder 104 can be connected to the IRQ ports of processor 120.

[0027] Processor 120 may include one or more processors, such as one or more image signal processors (ISPs) (including image processor 140), one or more host processors (not shown), and / or related processors. Figure 8 The computing system 800 may include one or more of any other type of processor 810 discussed. The processor 120 may include one or more digital signal processors (DSPs), central processing units (CPUs), graphics processing units (GPUs), and / or other types of processors.

[0028] In some implementations, processor 120 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) that includes image processor 140. In some cases, the chip may also include one or more input / output ports, a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, and connectivity components (e.g., Bluetooth). TM This includes, but is not limited to, Global Positioning System (GPS), and any combination thereof and / or other components. In some implementations, I / O ports can include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as Inter-Integrated Circuit 2 (I2C) interfaces, Inter-Integrated Circuit 3 (I3C) interfaces, Serial Peripheral Interface (SPI) interfaces, Serial General Purpose Input / Output (GPIO) interfaces, MIPI (such as MIPI CSI-2 PHY layer ports or interfaces), Advanced High Performance Bus (AHB) buses, any combination thereof and / or other input / output ports.

[0029] In some examples, processor 120 may include interrupt handler 125. In some examples, interrupt handler 125 may respond to an individual interrupt generated by decoder 104. For example, upon receiving IRQ 105 from sensor 102, processor 120 may signal programming engine 130 that the image from sensor 102 is ready for processing. In some cases, processor 120 may provide camera configuration data for sensor 102. In some cases, camera configuration data may include, but is not limited to, ISO, exposure time, aperture size, f / aperture, shutter speed, focus, and / or gain. In some examples, processor 120 may provide corresponding image processing configuration data to context buffer 132 of programming engine 130 associated with sensor 102. For example, image processing configuration data may include, but is not limited to, ISO, exposure time, aperture size, f / aperture (f / stop), shutter speed, focus, gain, and / or adjustments to contrast, brightness, saturation, sharpness, level, curves, and / or color.

[0030] In some cases, the processor 120 and / or interrupt handler 125 may detect that one or more of the sensors 102 have malfunctioned. For example, the processor may anticipate receiving an IRQ 105 associated with a specific sensor 102 at a specific time frame based on the provided camera configuration data. In some cases, if the IRQ 105 does not arrive within the expected time frame, the specific sensor 102 may be considered to be malfunctioning. In some cases, the interrupt handler 125 may perform one or more error handling operations to restore operation of the malfunctioning sensor. For example, in some cases, the interrupt handler 125 may instruct the programming engine 130 to refresh and / or invalidate the data corresponding to the malfunctioning sensor in the context buffer 132. In some cases, the malfunctioning sensor 102 may be reset and / or reprogrammed. In some implementations, the remaining sensors 102 may continue to operate while one or more error handling operations are performed for the malfunctioning sensor 102.

[0031] Programming engine 130 can be coupled to context buffer 132, which acquires and stores image processing configuration data associated with each sensor 102 until image processor 140 becomes available to process an image corresponding to the image processing configuration data. In an illustrative example, context buffer 132 can be implemented as a first-in, first-out (FIFO) buffer. Each of the context buffers 132 can be fed into programming module 135 of programming engine 130. In some cases, programming module 135 can communicate with image processor 140. For example, when image processor 140 becomes available (e.g., no image is currently being processed), the image processor can instruct it to be ready to process an image. In some cases, programming module 135 can determine which of the context buffers 132 contains the image processing configuration data for the next image to be processed by image processor 140 and can pass the image processing configuration data to image processor 140. In some cases, programming module 135 can select each context buffer 132 sequentially in a round-robin fashion. In some cases, a priority scheme can be used to prioritize images from one or more high-priority sensors. For example, images captured by sensors that can be analyzed by the vehicle’s computer vision system (e.g., a machine learning model) are more time-critical than images that are displayed to the vehicle’s driver.

[0032] In some cases, processor 120 can be a general-purpose processor (e.g., a CPU) that performs many other computational operations besides those of interrupt handler 125. For example, processor 120 can be used to execute... Figure 2 The operation of one or more of the vehicle computing system 250. In some cases, as the number of sensors 102 increases, using a general-purpose processor 120 to process the corresponding IRQ 105 may interfere with the operation of functions that rely on data (e.g., images) from the sensors 102 and / or other functions provided by the processor 120.

[0033] In some cases, dedicated hardware components, such as hardware interrupt handlers, can be provided to offload interrupt handling from the general-purpose processor 120. Figure 1B An example multi-sensor camera configuration 150 with an integrated hardware (HW) interrupt handler is shown. Figure 1B In the illustrative example, the interrupt handler can be incorporated into the programming engine / interrupt handler 160. As shown in the figure, by... Figure 1A The functionality provided by the interrupt handler 125 can be alternatively moved to Figure 1B The programming engine / interrupt handler 160. For example, the decoder 104 can provide IRQ 105 to the programming engine / interrupt handler 160.

[0034] In some cases, processor 121 can provide camera configuration data for sensor 102. In some cases, camera configuration data can include, but is not limited to, ISO, exposure time, aperture size, f / aperture, shutter speed, focus, and / or gain. In some examples, processor 121 can provide corresponding image processing configuration data to the context buffer 132 of the programming engine / interrupt handler 160 associated with sensor 102. For example, image processing configuration data can include, but is not limited to, ISO, exposure time, aperture size, f / aperture, shutter speed, focus, gain, and / or adjustments to contrast, brightness, saturation, sharpness, level, curves, and / or color.

[0035] In some examples, the programming / interrupt engine 165 can respond to individual interrupts generated by the decoder 104. For example, upon receiving an IRQ 105 from the sensor 102, the programming / interrupt engine 165 can signal to itself that the image from the sensor 102 is ready for processing.

[0036] In some cases, the programming / interrupt engine 165 can detect that one or more of the sensors 102 have malfunctioned. For example, the programming / interrupt engine 165 can anticipate receiving an IRQ 105 associated with a specific sensor 102 within a specific time frame. In some cases, if the IRQ 105 does not arrive within the expected time frame, it can be considered that the specific sensor 102 is exhibiting a malfunction. In some cases, the programming / interrupt engine 165 is capable of performing one or more error handling operations to restore the operation of the malfunctioning sensor 102. For example, in some cases, the programming / interrupt engine 165 is capable of refreshing and / or invalidating the data corresponding to the malfunctioning sensor in the context buffer 132. In some cases, the malfunctioning sensor 102 can be reset and / or reprogrammed. Figure 1B In this implementation, when the error handling operation is completed, the remaining sensors 102 may fail to continue operating. For example, if the programming / interrupt engine 165 is configured to provide image processing configuration data from the context buffer 132 in a cyclic manner, image processing operations of the remaining sensors can be blocked until the sensor exhibiting the error condition recovers its normal function. As a result, Figure 1B The configuration shown may not provide interference-free operation between sensors 102. In some cases, interference between sensors 102 due to sensor malfunction can pose a safety risk. In an illustrative example, if included... Figure 1B If one image sensor 102 of the multi-sensor camera configuration 150 of the autonomous driving system malfunctions, then as a result of the single point of failure of sensor 102, the autonomous driving system may be unable to perform autonomous driving functions.

[0037] Systems and techniques are needed for handling interruptions in sensor systems that include a large number of sensors (e.g., image sensors). For example, the ability to process images from one or more sensors to provide safety features for a vehicle. In some cases, failure to respond quickly to incoming sensor data can lead to inaccurate and / or unsafe maneuvering of the vehicle. For example, as the number of sensors increases, software-based interrupt handlers may become slow to respond and / or experience reduced performance. In some cases, a failure of a single sensor (e.g., sensor 102) can cause the reset of multiple sensors, even those that did not experience a failure. In some cases, independent operation of multiple sensors 102 following a non-interference principle may be preferred.

[0038] This document describes systems, devices, processes (also referred to as methods), and computer-readable media (collectively, “systems and techniques”) for interrupt handling in sensor systems (e.g., sensor systems comprising multiple sensors such as image sensors). For example, a hardware-based interrupt handler is capable of receiving interrupts from multiple sensors 102. In some cases, offloading interrupt handling from a processor (e.g., CPU) to a dedicated hardware component can free up computational resources. In some aspects, offloading interrupt handling to a dedicated hardware component can also provide reduced latency. In some examples, the systems and techniques described herein can further provide interference-free operation between sensors (e.g., sensor 102). For example, each of the multiple sensors can be associated with a context buffer of the interrupt handler. In some cases, if one of the sensors malfunctions, the interrupt handler can perform error handling for the specific sensor exhibiting the error while continuing to perform interrupt handling for the normally functioning sensors among the multiple sensors. Therefore, the systems and techniques described herein can be used to scalably provide interrupt handling for sensor systems comprising a large number of sensors.

[0039] Various aspects of this application will be described with reference to the accompanying drawings. Figure 2This is a block diagram illustrating an example of a vehicle computing system 250 for vehicle 204. Vehicle 204 is an example of a user equipment (UE) capable of communicating with a network (e.g., eNodeB, gNodeB, location beacon, location measurement unit, and / or other network entities) via a network interface (e.g., a Uu interface) and communicating with other UEs via a device-to-device direct interface using vehicle-to-everything (V2X) communication. As shown, the vehicle computing system 250 may include at least a power management system 251, a control system 252, an infotainment system 254, an intelligent transportation system (ITS) 255, one or more sensor systems 256, and a communication system 258. In some cases, the vehicle computing system 250 may include or be implemented using any type of processing device or system, such as one or more central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), application processors (APs), graphics processing units (GPUs), vision processing units (VPUs), neural network signal processors (NSPs), microcontrollers, dedicated hardware, any combination thereof, and / or other processing devices or systems.

[0040] Control system 252 can be configured to control the operation of one or more of the vehicle 204, power management system 251, computing system 250, infotainment system 254, ITS 255, and / or one or more other systems of vehicle 204 (e.g., braking system, steering system, safety systems other than ITS 255, cockpit systems, and / or other systems). In some examples, control system 252 can include one or more electronic control units (ECUs). ECUs can control one or more of the electrical systems or subsystems in the vehicle. Examples of specific ECUs that can be included as part of control system 252 include engine control module (ECM), powertrain control module (PCM), transmission control module (TCM), brake control module (BCM), central control module (CCM), central timing module (CTM), etc. In some cases, control system 252 can receive sensor signals from one or more sensor systems 256 and can communicate with other systems of vehicle computing system 250 to operate vehicle 204.

[0041] The vehicle computing system 250 also includes a power management system 251. In some implementations, the power management system 251 may include a power management integrated circuit (PMIC), a backup battery, and / or other components. In some cases, other systems of the vehicle computing system 250 may include one or more PMICs, a battery, and / or other components. The power management system 251 is capable of performing power management functions for the vehicle 204, such as managing the power supply for the computing system 250 and / or other parts of the vehicle. For example, the power management system 251 is capable of providing a stable power supply in response to power fluctuations, such as based on starting the vehicle's engine. In another example, the power management system 251 is capable of performing thermal monitoring operations, such as by checking the ambient temperature and / or transistor junction temperature. In another example, the power management system 251 is capable of performing certain functions based on the detection of a certain temperature level, such as causing a cooling system (e.g., one or more fans, an air conditioning system, etc.) to cool certain components of the vehicle computing system 250 (e.g., the control system 252, such as one or more ECUs), shutting down certain functions of the vehicle computing system 250 (e.g., limiting the infotainment system 254, such as by turning off one or more displays, disconnecting from a wireless network, etc.), and other functions.

[0042] The vehicle computing system 250 also includes a communication system 258. The communication system 258 is capable of including methods for transmitting data to a network (e.g., to a gNB or other network entity via a Uu interface) and / or to other UEs (e.g., via a PC5 interface, WiFi interface, Bluetooth). TM The interface and / or other wireless and / or wired interfaces transmit signals to another vehicle or UE and from the network (e.g., from a gNB or other network entity via the Uu interface) and / or from other UEs (e.g., via the PC5 interface, WiFi interface, Bluetooth). TM The software and hardware components that receive signals from another vehicle or UE via an interface and / or other wireless and / or wired interfaces. For example, the communication system 258 is configured to receive signals via any suitable wireless network (e.g., 3G network, 4G network, 5G network, WiFi network, Bluetooth). TM The communication system 258 wirelessly transmits and receives information via a network and / or other networks. It includes various components or devices for performing wireless communication functions.

[0043] In some cases, the communication system 258 may also include one or more wireless interfaces for transmitting and receiving wireless communications (e.g., including one or more transceivers and one or more baseband processors for each wireless interface), one or more wired interfaces for performing communication via one or more hardwired connections (e.g., serial interfaces such as Universal Serial Bus (USB) inputs, Lightning connectors, and / or other wired interfaces), and / or other components that enable the vehicle 204 to communicate with a network and / or other UEs.

[0044] The vehicle computing system 250 may also include an infotainment system 254 capable of controlling content and one or more output devices of the vehicle 204 capable of outputting content. The infotainment system 254 may also be referred to as an in-vehicle infotainment (IVI) system or an in-vehicle entertainment (ICE) system. Content may include navigation content, media content (e.g., video content, music or other audio content and / or other media content), and other content. One or more output devices may include one or more graphical user interfaces, one or more displays, one or more speakers, one or more extended reality devices (e.g., VR, AR, and / or MR headsets), one or more haptic feedback devices (e.g., one or more devices configured to vibrate the seats, steering wheel, and / or other parts of the vehicle 204), and / or other output devices.

[0045] In some examples, the vehicle computing system 250 may include an Intelligent Transportation System (ITS) 255. In some examples, the ITS 255 can be used to implement V2X communication. For example, the ITS stack of the ITS 255 can generate V2X messages based on information from the application layer of the ITS. In some cases, the application layer can determine whether certain conditions have been met for generating messages for use by the ITS 255 and / or for generating messages to be sent to other vehicles (for V2V communication), pedestrian UEs (for V2P communication), and / or infrastructure systems (for V2I communication). In some cases, the communication system 258 and / or the ITS 255 can obtain Vehicle Access Network (CAN) information (e.g., from other components of the vehicle via the CAN bus). In some examples, the communication system 258 (e.g., TCU NAD) can obtain CAN information via the CAN bus and can send the CAN information to the ITS stack. The CAN information can include vehicle-related information such as the vehicle's heading, speed, interruption information, and other information. CAN information can be provided to the ITS 255 continuously or periodically (e.g., every 1 millisecond (ms), every 10 ms, etc.).

[0046] The ITS 255 can use CAN information to determine the conditions for generating a message based on safety-related applications and / or other applications (including applications related to road safety, traffic efficiency, infotainment, business, and / or other applications). In an illustrative example, the ITS 255 can perform lane change assistance or negotiation. For example, using CAN information, the ITS 255 can determine that the driver of vehicle 204 is attempting to change lanes from the current lane to an adjacent lane (e.g., based on a turn signal being activated, based on the user turning or turning into an adjacent lane, etc.). Based on determining that vehicle 204 is attempting to change lanes, the ITS 255 can determine that lane change conditions have been met, where the lane change conditions are associated with messages to be sent to other vehicles near the vehicle in the adjacent lane. The ITS 255 can trigger the ITS stack to generate one or more messages for transmission to other vehicles, which can be used to negotiate a lane change with other vehicles. Other examples of applications include forward collision warning, automatic emergency braking, lane departure warning, pedestrian avoidance or protection (e.g., when a pedestrian is detected near vehicle 204, such as based on V2P communication with the user's UE), traffic sign recognition, etc.

[0047] The ITS 255 is capable of using any suitable protocol to generate messages (e.g., V2X messages). Examples of protocols that can be used by the ITS 255 include one or more Society of Automotive Engineers (SAE) standards, such as SAE J2735, SAE J2945, SAE J3161 and / or other standards, the entire contents of which are incorporated herein by reference and used for all purposes.

[0048] The security layer of ITS 255 can be used to securely sign messages from the ITS stack, where the messages are sent to and verified by other UEs (such as other vehicles, pedestrian UEs, and / or infrastructure systems) configured for V2X communication. The security layer can also verify messages received from such other UEs. In some implementations, the signing and verification process can be based on the vehicle's security context. In some examples, the security context may include one or more encryption-decryption algorithms, a public and / or private key used to generate the signature using the encryption-decryption algorithms, and / or other information. For example, each ITS message generated by the ITS stack can be signed by the security layer. The signature can be derived using the public key and the encryption-decryption algorithm. The vehicle, pedestrian UE, and / or infrastructure system receiving the signed message can verify the signature to ensure the message originates from an authorized vehicle. In some examples, one or more encryption-decryption algorithms can include one or more symmetric encryption algorithms (e.g., Advanced Encryption Standard (AES), Data Encryption Standard (DES), and / or other symmetric encryption algorithms), one or more asymmetric encryption algorithms using public and private keys (e.g., Rivest-Shamir-Adleman (RSA) and / or other asymmetric encryption algorithms), and / or other encryption-decryption algorithms.

[0049] The computing system 250 also includes one or more sensor systems 256 (e.g., a first sensor system to an Nth sensor system, where N is a value equal to or greater than 0). When multiple sensor systems are included, the sensor systems 256 can include different types of sensor systems that can be arranged on or in different parts of the vehicle 204. The sensor systems 256 can include one or more image sensors 259. In an illustrative example, one or more image sensors 259 can be included in a multi-camera system (e.g., Figure 3 Multi-sensor camera configuration 300, Figure 4 The multi-sensor camera configuration 400 is included. In some cases, the vehicle computing system 250 may include an interrupt handler 257 (e.g., Figure 1A Interrupt handler 125 Figure 1B Programming engine / interrupt handler 160 Figure 3 Programming engine / interrupt handler 330 Figure 4 The first programming engine / interrupt handler 430 and / or the second programming engine / interrupt handler 431). In some cases, the interrupt handler(s) 257 can be used to manage the processing of sensor data from the sensor system(s) 256 according to the systems and techniques described herein.

[0050] The (multiple) sensor system 256 may also include one or more optical detection and ranging (LIDAR) sensor systems, radio detection and ranging (RADAR) sensor systems, electromagnetic detection and ranging (EmDAR) sensor systems, sound navigation and ranging (SONAR) sensor systems, sound detection and ranging (SODAR) sensor systems, global navigation satellite system (GNSS) receiver systems (e.g., one or more global positioning system (GPS) receiver systems), accelerometers, gyroscopes, inertial measurement units (IMUs), infrared sensor systems, laser rangefinder systems, ultrasonic sensor systems, infrasound sensor systems, microphones, any combination thereof, and / or other sensor systems. It should be understood that any number of sensors or sensor systems can be included as part of the computing system 250 of the vehicle 204.

[0051] Although the vehicle computing system 250 is shown as including certain components and / or systems, those skilled in the art will understand that the vehicle computing system 250 is capable of including more than [other components and / or systems]. Figure 2 The components shown may include more or fewer components. For example, the vehicle computing system 250 may also include one or more input devices and one or more output devices (not shown). In some implementations, the vehicle computing system 250 may also include (e.g., as part of or separate from the control system 252, infotainment system 254, communication system 258 and / or (multiple) sensor systems 256) at least one processor and at least one memory having computer-executable instructions that are executed by the at least one processor. The at least one processor is communicatively and / or electrically connected (referred to as “coupled to” or “communically coupled”) to the at least one memory. The at least one processor may include, for example, one or more microcontrollers, one or more central processing units (CPUs), one or more field-programmable gate arrays (FPGAs), one or more graphics processing units (GPUs), one or more application processors (e.g., for running or executing one or more software applications) and / or other processors. At least one memory may include, for example, read-only memory (ROM), random access memory (RAM) (e.g., static RAM (SRAM)), electrically erasable programmable read-only memory (EEPROM), flash memory, one or more buffers, one or more databases, and / or other memories. It is capable of executing computer-executable instructions stored in or on at least the memory to perform one or more of the functions or operations described herein.

[0052] Figure 3This is a block diagram illustrating a multi-sensor camera configuration 300 including HW interrupt handling, wherein HW interrupt handling can maintain the operation of some sensors 102 while other sensors 102 exhibit errors. Therefore, in some cases, Figure 3 The multi-sensor camera configuration 300 is capable of providing interrupt handling unaffected by erroneous handling of one or more sensors 302 exhibiting errors. As shown, the multi-sensor camera configuration 300 includes multiple sensors 302, multiple decoders 304, a programming engine / interrupt handler 330, a processor 321, and an image processor 340. Figure 3 In the illustrative example, sensor 302 is capable of being similar to Figure 1A and Figure 1B The sensor 102 performs similar functions. Figure 1A and Figure 1B The sensor 102 has the same functions. The image processor 340 is capable of similar functions. Figure 1A and Figure 1B Image processor 140 and performs similar functions Figure 1A and Figure 1B The image processor 140 has the same functions. Similarly, the decoder 304 is capable of similar functions. Figure 1A and Figure 1B Decoder 104 and performs similar functions Figure 1A and Figure 1B The function of decoder 104. For example... Figure 3 As shown, the decoder 304 can provide IRQ 305 to the IRQ port of the programming engine / interrupt handler 330.

[0053] Processor 321 can be configured to provide camera configuration data from sensor 302. In some cases, camera configuration data may include, but is not limited to, ISO, exposure time, aperture size, f / aperture, shutter speed, focus, and / or gain. In some examples, processor 321 can provide corresponding image processing configuration data to the context buffer 332 of programming engine / interrupt handler 330. For example, image processing configuration data may include, but is not limited to, ISO, exposure time, aperture size, f / aperture, shutter speed, focus, gain, and / or adjustments to contrast, brightness, saturation, sharpness, level, curves, and / or color. In some cases, processor 321 can perform other operations, such as... Figure 2 Any operation of the vehicle computing system 250.

[0054] The programming engine / interrupt handler 330 includes a context buffer 332, a programming module 335, multiple interrupt handlers 350, and a context engine 360. In the illustrative example, the context buffer 332 can be similar to... Figure 1A and Figure 1BThe context buffer 132 and perform similar actions Figure 1A and Figure 1B The context buffer 132 functions similarly. Similarly, the programming module 335 can function similarly to... Figure 1A and Figure 1B The programming module 135 and performs similar functions. Figure 1A and Figure 1B The functionality of programming module 135. For example... Figure 3 As shown, each sensor 302 can be associated with a separate interrupt handler 350 included in the programming engine / interrupt handler 330. For example, if an error occurs on sensor 302 labeled sensor 1, the corresponding interrupt handler 351 can perform error handling operations without interrupting the operation of the remaining sensors 302. For example, interrupt handler 351 can signal a non-blocking wait for the corresponding sensor 1. In some examples, based on the non-blocking wait, programming module 335 can skip sensor 302 in a sequence of providing image processing configuration data from context buffer 332 to image processor 340. In some cases, error handling operations can include resetting and / or reprogramming the sensor 302 exhibiting the error.

[0055] In some cases, interrupt handler 350 and / or context engine 360 ​​can refresh and / or invalidate data contained in context buffer 332 corresponding to sensor 302 exhibiting an error. In some cases, refreshing and / or invalidating data contained in context buffer 332 can prevent image processor 340 from processing erroneous sensor data.

[0056] Figure 4 A block diagram of a multi-sensor camera configuration 400 including multiple interrupt handlers is shown. The multi-sensor camera configuration 400 includes a first set of sensors 402, a first set of decoders 404, a second set of sensors 403, a second set of decoders 407, a processor 421, a first programming engine / interrupt handler 430, a second programming engine / interrupt handler 431, a first image processor 440, and a second image processor 441.

[0057] Processor 421 can be configured to provide camera configuration data for the first set of sensors 402 and the second set of sensors 403. In some cases, the camera configuration data may include, but is not limited to, ISO, exposure time, aperture size, f / aperture, shutter speed, focus, and / or gain. In some examples, processor 421 can provide corresponding image processing configuration data to the context buffer of the first programming engine / interrupt handler 430 associated with the first set of sensors 402 (e.g., ...). Figure 3The context buffer 332). Similarly, the processor 421 can provide the corresponding image processing configuration data to the context buffer of the second programming engine / interrupt handler 431 associated with the second set of sensors 403 (e.g., context buffer 332). Figure 3 (Context buffer 332). For example, image processing configuration data can include, but is not limited to, adjustments to ISO, exposure time, aperture size, f / aperture, shutter speed, focus, gain, and / or contrast, brightness, saturation, sharpness, level, curves, and / or color.

[0058] exist Figure 4 In the illustrative example, the first programming engine / interrupt handler 430 and the second programming engine / interrupt handler 431 can be similar to Figure 3 The programming engine / interrupt handler 330 and performs similar functions. Figure 3 The programming engine / interrupt handler 430 functions as follows. As shown, the first programming engine / interrupt handler 430 can obtain IRQ 405 from the first set of sensors 402 / first set of decoders 404. The first programming engine / interrupt handler 430 can provide the first image processor 440 with image processing configuration data associated with the first set of sensors 402 / first set of decoders 404. Similarly, the second programming engine / interrupt handler 431 can obtain IRQ 405 from the second set of sensors 403 / second set of decoders 407 and provide the second image processor 441 with image processing configuration data associated with the second set of sensors 403 / second set of decoders 407. In some cases, the processor 421 can perform other operations, such as Figure 2 Any operation of the vehicle computing system 250.

[0059] In an illustrative example, both the first programming engine / interrupt handler 430 and the second programming engine / interrupt handler 431 are capable of performing interrupt handling for an integer N sensors. Therefore, the multi-sensor camera configuration 400 can provide interrupt handling for 2*N sensors. In some cases, additional scaling for additional sensors can be provided by additional programming engines / interrupt handlers and / or image processors. Although Figure 4 The example shows that the first programming engine / interrupt handler 430 and the second programming engine / interrupt handler 431 provide interrupt handling for an equal number of sensors, but in some cases, without departing from the scope of this disclosure, two or more interrupt programming engines / interrupt handlers are able to provide interrupt handling for different numbers of sensors.

[0060] As described above, the systems and techniques described herein can be used to provide interrupt handling for sensor systems with a large number of sensors. These systems and techniques can provide interference-free operation between sensors exhibiting errors and those appearing to operate without errors. Conversely, software-based interrupt handlers utilizing general-purpose processors (e.g., CPUs) are less suitable for handling large numbers of sensors (e.g., ...). Figure 1A The scaling is achieved by increasing the number of sensors (102). This scaling also applies to IRQs (e.g., Figure 1A With an increase in the number of IRQs (105), general-purpose processors may fail to respond to IRQs in a timely manner, and / or other functions performed by general-purpose processors may be affected. In some cases, multi-sensor systems, including those with HW-based interrupt handlers, may fail to provide interference-free operation. For example, in some situations, HW-based interrupt handlers may become blocked from handling IRQs of sensors that do not exhibit errors.

[0061] Conversely, the systems and techniques described herein offer the benefits of offloading HW-based interrupt handling from general-purpose processors, while providing a non-interference-free experience. In some cases, a separate interrupt handler can be provided for each of multiple sensors. When a sensor exhibits an error, the corresponding interrupt handler can perform error handling operations. For example, the interrupt handler can apply non-blocking waits to programming modules (e.g., Figure 3 (Programming module 335). In some cases, the interrupt handler can refresh and / or invalidate the data in the context buffer 332 corresponding to the sensor exhibiting the error. In some cases, the systems and techniques described herein can scale with the number of sensors by providing multiple HW-based interrupt handlers operating in parallel.

[0062] Figure 5 This is a flowchart illustrating an example of a process 500 for interrupt handling. At box 502, process 500 includes a first interrupt handler (e.g., Figure 3 The first context buffer associated with the interrupt handler 350 (e.g., Figure 3 The context buffer 332) obtains the information from the first image sensor (e.g., sensor 102 of FIG1). Figure 3 The first image processing configuration data is associated with sensor 302. In some cases, the first image processing configuration data includes at least one of exposure settings, gain, resolution, pixel configuration, motion indication, or dynamic range settings. In some examples, the first context buffer and the second context buffer are first-in-first-out (FIFO) buffers.

[0063] At box 504, process 500 includes a second interrupt handler (e.g., Figure 3The second context buffer associated with the interrupt handler 350 (e.g., Figure 3 The context buffer 332) obtains a second image sensor (e.g., sensor 102 of FIG1) that is different from the first image sensor. Figure 3 The second image processing configuration data associated with sensor 302.

[0064] At box 506, process 500 includes obtaining an error indication associated with the first image sensor at the first interrupt handler.

[0065] At block 508, process 500 includes an error handling operation performed by a first interrupt handler based on an error indication associated with the first image sensor. In some examples, the error handling operation includes at least one of refreshing a first context buffer or invalidating first image processing configuration data in the first context buffer. In some examples, the error handling operation includes resetting the first image sensor. In some cases, the error handling operation includes a non-blocking wait associated with at least one of a second image sensor or a second context buffer. In some examples, during the non-blocking wait, the second image sensor remains operational while the first image sensor or at least one of the first context buffers associated with the first interrupt handler is inoperable.

[0066] At block 510, process 500 includes obtaining an IRQ from the second image sensor at the second interrupt handler during a time window interval between obtaining an error indication associated with the first image sensor and the completion time of the first error handling operation. In some aspects, the IRQ corresponds to the completion of an image capture operation performed by the second image sensor. The image capture operation includes storing an image in memory.

[0067] At block 512, process 500 includes an interrupt handler outputting second image processing configuration data to the image processor based on the obtained IRQ.

[0068] In some cases, process 500 includes obtaining second image processing configuration data at an image processor and processing an image associated with the second image processing configuration data based on the second image processing configuration data.

[0069] In some cases, process 500 includes interaction with a third interrupt handler (e.g., Figure 3 The third context buffer associated with the interrupt handler 350 (e.g., Figure 3 The context buffer 332) obtains the information from the third image sensor (e.g., sensor 102 in Figure 1). Figure 2The process 500 includes obtaining additional IRQs from the third image sensor (302) associated with the second image processing configuration data. In some examples, process 500 includes obtaining an indication from the image processor that processing of the image associated with the second image processing configuration data has been completed. In some aspects, process 500 includes outputting the third image processing configuration data to the image processor by the third interrupt handler based on the obtained additional IRQs.

[0070] In some examples, the multiple image sensors include a first image sensor and a second image sensor. In some cases, the multiple interrupt handlers include a first interrupt handler and a second interrupt handler. In some implementations, the interrupt handlers among the multiple interrupt handlers are configured to process IRQs from the multiple image sensors in a cyclical order.

[0071] In some examples, the multiple image sensors include a first image sensor and a second image sensor. In some cases, the multiple interrupt handlers include a first interrupt handler and a second interrupt handler. In some implementations, the interrupt handlers among the multiple interrupt handlers are configured to process IRQs from the multiple image sensors in priority order. In some aspects, a high-priority group includes at least one image sensor among the multiple image sensors, and a low-priority group includes at least one different image sensor among the multiple image sensors.

[0072] In some examples, the processes described herein (e.g., process 500 and / or other processes described herein) can be executed by a computing device or apparatus. In one example, one or more processes can be executed by... Figure 8 The vehicle computing system 250 performs the operation. In another example, one or more processes can be performed by... Figure 8 The computing system 800 shown is used to perform this operation. For example, it has... Figure 5 The computing device of the computing system 800 shown can include components of the vehicle computing system 250 and can realize... Figure 6 The operation of process 500 and / or other processes described herein.

[0073] Computing devices can include any suitable device, such as a vehicle or vehicle computing device (e.g., a driver monitoring system (DMS) for a vehicle), mobile device (e.g., a mobile phone), desktop computing device, tablet computing device, wearable device (e.g., VR headset, AR headset, AR glasses, connected watch or smartwatch or other wearable device), server computer, robotic device, television, and / or any other computing device with the resource capability to perform the processes described herein (including process 500 and / or other processes described herein). In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or multiple other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or multiple other components. The network interface may be configured to transmit and / or receive Internet Protocol (IP) based data or other types of data.

[0074] Components of a computing device can be implemented in a circuit. For example, a component can include electronic circuitry or other electronic hardware and / or can be implemented using electronic circuitry or other electronic hardware, wherein the electronic circuitry or other electronic hardware can include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or can include computer software, firmware, or any combination thereof and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0075] Process 500 is shown as a logic flowchart, whose operations represent a series of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the stated operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0076] Furthermore, process 500 and / or other processes described herein can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, hardware, or a combination thereof. As described above, the code can be stored, for example, on a computer-readable or machine-readable storage medium in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0077] As described above, various aspects of this disclosure can utilize machine learning models or systems. Figure 7 This is an illustrated example of a deep learning neural network 600 capable of implementing the machine learning-based feature extraction and / or activity recognition (or classification) described above. Input layer 620 includes input data. In an illustrative example, input layer 620 may include data representing pixels of an input video frame. Neural network 600 includes multiple hidden layers 622a, 622b through 622n. Hidden layers 622a, 622b through 622n include a number of "n" hidden layers, where "n" is an integer greater than or equal to 1. The number of hidden layers can be as many as required for a given application. Neural network 600 also includes an output layer 621, which provides the output produced by the processing performed by hidden layers 622a, 622b through 622n. In an illustrated example, output layer 621 may provide a classification of objects in the input video frame. The classification may include categories recognizing the type of activity (e.g., looking up, looking down, eyes closed, yawning, etc.).

[0078] Neural network 600 is a multi-layered neural network with interconnected nodes. Each node can represent a piece of information. The information associated with a node is shared between different layers, and each layer retains the information as it is processed. In some cases, neural network 600 can include a feedforward network, in which case there is no feedback connection where the network's output is fed back into itself. In some cases, neural network 600 can include a recurrent neural network, which can have loops that allow information to be carried across nodes when reading input.

[0079] Information can be exchanged between nodes through node-to-node interconnections between layers. Nodes in input layer 620 can activate a set of nodes in the first hidden layer 622a. For example, as shown, each input node in input layer 620 is connected to each node in the first hidden layer 622a. Nodes in the first hidden layer 622a can transform the information of each input node by applying an activation function to the input node information. The information derived from the transformation can then be passed to and activated by nodes in the next hidden layer 622b, where each node in the next hidden layer 622b can perform its own specified function. Example functions include convolution, upsampling, data transformation, and / or any other suitable function. The output of hidden layer 622b can then activate nodes in the next hidden layer, and so on. Finally, the output of hidden layer 622n can activate one or more nodes in output layer 621, where the output is provided. In some cases, although a node in neural network 600 (e.g., node 626) is shown as having multiple output lines, the node has a single output and all lines shown as outputs from the node represent the same output value.

[0080] In some cases, each node or the interconnection between nodes can have weights, where said weights are a set of parameters derived from the training of the neural network 600. Once the neural network 600 is trained, it can be called a trained neural network, which can be used to classify one or more activities. For example, the interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnection can have tunable numerical weights that can be tuned (e.g., based on the training dataset), allowing the neural network 600 to adapt to the input and learn as more and more data is processed.

[0081] The neural network 600 is pre-trained to process features from the data in the input layer 620 using different hidden layers 622a, 622b to 622n, so as to provide an output through the output layer 621. In an example where the neural network 600 is used to identify the activity being performed by the driver in a frame, as described above, the neural network 600 can be trained using training data that includes both frames and labels. For example, training frames can be input into the network, each training frame having a label indicating the features in the frame or a label indicating the activity category in each frame. In an example using object classification for illustrative purposes, the training frame can include an image numbered 2, in which case the image label can be [0 0 1 0 0 0 0 0 0 0].

[0082] In some cases, the neural network 600 can use a training process called backpropagation to adjust the weights of its nodes. As described above, the backpropagation process can include forward pass, loss function, backpropagation, and weight update. Forward pass, loss function, backpropagation, and parameter update are performed for each training iteration. This process can be repeated a certain number of iterations for each set of training images until the neural network 600 is sufficiently trained so that the layer weights are accurately tuned.

[0083] For an example of recognizing objects in a frame, the forward pass can include passing a training frame through a neural network 600. The weights are initially randomized before the neural network 600 is trained. As an illustrative example, a frame can include a numerical array representing pixels of an image. Each number in the array can include a value from 0 to 255, describing the pixel intensity at that location in the array. In one example, the array can include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and three color components (such as red, green, and blue, or luminance and two chromaticity components).

[0084] As mentioned above, for the first training iteration of a neural network 600, the output may include values ​​that do not give any particular class preference due to the random selection of weights during initialization. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each different class can be equal or at least very similar (e.g., for ten possible classes, each class could have a probability value of 0.1). Using the initial weights, the neural network 600 cannot determine low-level features and therefore cannot accurately determine what the object's classification might be. Loss functions can be used to analyze errors in the output. Any suitable loss function definition can be used, such as cross-entropy loss. Another example of a loss function includes mean squared error (MSE), which is defined as... The loss can be set to equal to The value of .

[0085] For the first training image, the loss (or error) will be high because the actual value will be significantly different from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training label. The Neural Network 600 can perform backpropagation by determining which inputs (weights) contribute most to the network's loss and can adjust the weights to reduce and eventually minimize the loss. The derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight at a specific layer) can be calculated to determine the weights that contribute most to the network's loss. After the derivative is calculated, weight updates can be performed by updating all the weights of the filter. For example, weights can be updated so that they change in the opposite direction of the gradient. Weight updates can be expressed as... Where w represents the weight, w iLet represent the initial weights, and η represent the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates, and a lower value indicates smaller weight updates.

[0086] Neural Network 600 can include any suitable deep network. An example includes a Convolutional Neural Network (CNN), which consists of an input layer and an output layer, with multiple hidden layers between them. The hidden layers of a CNN consist of a series of convolutional, non-linear, pooling (for downsampling), and fully connected layers. Neural Network 600 can also include any other deep network besides CNNs, such as autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), etc.

[0087] Figure 7 This is an illustrative example of a Convolutional Neural Network (CNN) 700. The input layer 720 of the CNN 700 includes data representing an image or frame. For example, the data can include a numerical array representing pixels of an image, where each number in the array includes a value from 0 to 255 describing the pixel intensity at that location in the array. Using the previous example above, the array can include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luminance and two chrominance components, etc.). The image can be passed through a convolutional hidden layer 722a, an optional non-linear activation layer, a pooling hidden layer 722b, and a fully connected hidden layer 722c to obtain an output at the output layer 724. While in Figure 7 Only one of each hidden layer is shown in the image, but those skilled in the art will recognize that a CNN 700 can include multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers. As previously described, the output can indicate a single category of an object, or can include probabilities that best describe the category of an object in an image.

[0088] The first layer of CNN 700 is a convolutional hidden layer 722a. Convolutional hidden layer 722a analyzes the image data input to layer 720. Each node in convolutional hidden layer 722a is connected to a region of the input image, called the receptive field. Convolutional hidden layer 722a can be thought of as one or more filters (each filter corresponding to a different activation or feature map), where each convolutional iteration of the filter is a node or neuron in convolutional hidden layer 722a. For example, the region of the input image covered by the filter at each convolutional iteration will be the filter's receptive field. In an illustrative example, if the input image comprises a 28×28 array and each filter (and its corresponding receptive field) is a 5×5 array, then there will be 24×24 nodes in convolutional hidden layer 722a. Each connection between a node and its receptive field learns weights and, in some cases, learns an overall bias, such that each node learns to analyze its specific local receptive field in the input image. Each node in hidden layer 722a will have the same weights and biases (called shared weights and shared biases). For example, the filter has an array of weights (numbers) and the same depth as the input. For the video frame example, the filter would have a depth of 3 (based on the three color components of the input image). An illustrative example of the filter array size is 5×5×3, corresponding to the size of the receptive field of a node.

[0089] The convolutional property of the convolutional hidden layer 722a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 722a can start at the top left corner of the input image array and can convolve around the input image. As described above, each convolutional iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 722a. At each convolutional iteration, the filter value is multiplied by the corresponding number of original pixel values ​​of the image (e.g., multiplying a 5×5 filter array by a 5×5 array of input pixel values ​​at the top left corner of the input image array). The multiplications from each convolutional iteration can be summed together to obtain the sum of that iteration or node. Next, the process continues at the next position in the input image based on the receptive field of the next node in the convolutional hidden layer 722a. For example, the filter can move to the next receptive field with a step size (called stride). The stride can be set to 1 or other suitable amounts. For example, if the stride is set to 1, the filter will move 1 pixel to the right at each convolutional iteration. Processing the filter at each unique location of the input convolution produces a number representing the filter result at that location, resulting in a sum value being determined for each node of the convolutional hidden layer 722a.

[0090] The mapping from the input layer to the convolutional hidden layer 722a is called an activation map (or feature map). An activation map includes the value of each node representing the filter result at each location on the input volume. Activation maps can comprise arrays, where the array includes various sums of values ​​generated by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will comprise a 24×24 array. The convolutional hidden layer 722a can include several activation maps to identify multiple features in the image. Figure 7 The example shown includes three activation maps. Using the three activation maps, the convolutional hidden layer 722a is able to detect three different kinds of features, where each feature is detectable across the entire image.

[0091] In some examples, it is possible to apply nonlinear hidden layers after convolutional hidden layer 722a. Nonlinear layers can be used to introduce nonlinearity into a system that has already computed linear operations. An illustrative example of a nonlinear layer is the Rectified Linear Unit (ReLU) layer. The ReLU layer applies the function f(x) = max(0, x) to all values ​​in the input convolution, which changes all negative activations to 0. Therefore, ReLU can increase the nonlinearity of CNN 700 without affecting the receptive field of convolutional hidden layer 722a.

[0092] Pooling hidden layer 722b can be applied after convolutional hidden layer 722a (and, when used, after a non-linear hidden layer). Pooling hidden layer 722b is used to simplify the information in the output of convolutional hidden layer 722a. For example, pooling hidden layer 722b can take each activation map output from convolutional hidden layer 722a and use a pooling function to generate a compressed activation map (or feature map). Max pooling is an example of a function performed by a pooling hidden layer. Pooling hidden layer 722a can use other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. Pooling functions (e.g., max pooling filters, L2 norm filters, or other suitable pooling filters) are applied to each activation map included in convolutional hidden layer 722a. Figure 8 In the example shown, three pooling filters are used to convolve the three activation maps in the hidden layer 722a.

[0093] In some examples, max pooling can be used by applying a max pooling filter (e.g., of size 2×2) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from the convolutional hidden layer 722a. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer is able to summarize a region of 2×2 nodes from the previous layer (where each node is a value in the activation map). For example, the four values ​​(nodes) in the activation map will be analyzed by the 2×2 max pooling filter at each iteration of the filter, where the maximum value from the four values ​​is output as the "max" value. If such a max pooling filter is applied to the activation filter of the convolutional hidden layer 722a with a dimension of 24×24 nodes, the output from the pooling hidden layer 722b will be an array of 12×12 nodes.

[0094] In some examples, it is also possible to use an L2 norm pooling filter. An L2 norm pooling filter involves calculating the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (instead of calculating the maximum value as done in max pooling) and using the calculated value as the output.

[0095] Intuitively, pooling functions (e.g., max pooling, L2-norm pooling, or other pooling functions) determine whether a given feature is found anywhere in a region of an image, discarding the exact location information. This can be done without affecting the results of feature detection because once a feature is found, its exact location is less important than its approximate location relative to other features. Max pooling (and other pooling methods) offers the benefit of having far fewer pooling features, thus reducing the number of parameters required in later layers of the CNN 700.

[0096] The final connection in the network is a fully connected layer, which connects each node from the pooling hidden layer 722b to each of the output nodes in the output layer 724. Using the example above, the input layer comprises 28×28 nodes encoding the pixel intensity of the input image, the convolutional hidden layer 722a comprises 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for filtering) to three activation maps, and the pooling hidden layer 722b comprises a layer of 3×12×12 hidden feature nodes based on applying a max-pooling filter to a 2×2 region across each of the three feature maps. Extending this example, the output layer 724 can comprise ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 722b is connected to each node of the output layer 724.

[0097] The fully connected layer 722c takes the output of the previous pooling hidden layer 722b (which should represent the activation map of high-level features) and determines the features most relevant to a particular class. For example, the fully connected layer 722c can determine the high-level features most relevant to a particular class and can include the weights (nodes) of the high-level features. The product between the weights of the fully connected layer 722c and the pooling hidden layer 722b can be computed to obtain the probabilities of different classes. For example, if CNN 700 is being used to predict whether an object in a video frame is a person, there will be high values ​​in the activation map representing the high-level features of a person (e.g., two legs, the face at the top of the object, two eyes at the top left and top right of the face, the nose in the middle of the face, the mouth at the bottom of the face, and / or other features common to people).

[0098] In some examples, the output from output layer 724 can include an M-dimensional vector (M=10 in the previous example). M indicates the number of classes the CNN 700 has to choose from when classifying objects in an image. Other example outputs can also be provided. Each number in the M-dimensional vector can represent the probability that an object belongs to a certain class. In an illustrative example, if the 10-dimensional output vector representing objects in ten different classes is [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates a 5% probability that the image is an object in the third class (e.g., a dog), an 80% probability that the image is an object in the fourth class (e.g., a person), and a 15% probability that the image is an object in the sixth class (e.g., a kangaroo). The probability of a class can be considered as the confidence level that an object is part of that class.

[0099] Figure 8 This is a diagram illustrating an example of a system used to implement certain aspects of this technology. In particular, ​ An example of a computing system 800 is shown, which can be, for example, any computing device constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 805. Connection 805 can be a physical connection using a bus, or a direct connection to processor 810, such as in a chipset architecture. Connection 805 can also be a virtual connection, a networking connection, or a logical connection.

[0100] In some embodiments, the computing system 800 is a distributed system, wherein the functions described herein can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a plurality of such components, wherein each component performs some or all of the functions described for that component. In some embodiments, a component may be a physical or virtual device.

[0101] Example system 800 includes at least one processing unit (CPU or processor) 810 and a connection 805 that couples various system components, including system memory 815 (such as read-only memory (ROM) 820 and random access memory (RAM) 825), to processor 810. Computing system 800 is capable of including a cache 812 of high-speed memory that is directly connected to, adjacent to, or integrated into processor 810.

[0102] Processor 810 can include any general-purpose processor and hardware or software services, such as services 832, 834, and 836 stored in storage device 830, which are configured to control processor 810 and dedicated processors, wherein software instructions are incorporated into the actual processor design. Processor 810 can essentially be a completely independent computing system, containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0103] To enable user interaction, the computing system 800 includes an input device 845 capable of representing any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, and voice input. The computing system 800 may also include an output device 835, which can be one or more of multiple output mechanisms. In some cases, a multimodal system enables users to provide multiple types of input / output to communicate with the computing system 800. The computing system 800 may include a communication interface 840, which typically controls and manages user input and system output. The communication interface may use wired and / or wireless transceivers to perform or facilitate the reception and / or transmission of wired or wireless communications, including those transceivers using the following: audio jack / plug, microphone jack / plug, Universal Serial Bus (USB) port / plug, Apple® Lightning® port / plug, Ethernet port / plug, fiber optic port / plug, proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communication (DSRC) wireless signal transmission, and 802.11. The communication interface 840 may include Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), global microwave access interoperability (WiMAX), infrared (IR) wireless signal transmission, public switched telephone network (PSTC) signal transmission, integrated services digital network (ISDN) digital transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or a combination of the above. The communication interface 840 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing system 800 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There are no restrictions on operation on any particular hardware layout, so the basic features here can be easily replaced as improved hardware or firmware layouts are developed.

[0104] Storage device 830 can be a non-volatile and / or non-transitory and / or computer-readable storage device, and can be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape cassettes, flash memory cards, solid-state storage devices, digital universal disks, cassette tapes, floppy disks, flexible disks, hard disks, magnetic tapes, magnetic stripes / strips, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, optical disc read-only memory (CD-ROM), rewritable optical disc (CD), digital video disc (DVD), Blu-ray disc (BDD), holographic disc, another optical medium, secure digital (SD) cards, microsecure digital (microSD) cards, memory Stick® cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette tape and / or combinations thereof.

[0105] Storage device 830 may include software services, servers, etc., which cause the system to perform functions when the code defining such software is executed by processor 810. In some embodiments, hardware services that perform specific functions may include software components stored in a computer-readable medium in conjunction with necessary hardware components (such as processor 810, connection 805, output device 835, etc.) to perform said functions.

[0106] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include, but not include, non-transitory media capable of storing data, carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact optical discs (CDs) or digital versatile optical discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or sent using any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0107] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0108] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks, which include devices, device components, steps, or routines in a method embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0109] The foregoing may describe individual embodiments as processes or methods, which are depicted as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0110] The processes and methods according to the examples above can be implemented using computer-executable instructions stored on or otherwise obtained from a computer-readable medium. Such instructions can include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a specific function or group of functions. Parts of the computer resources used are accessible via a network. The computer-executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information, and / or information created during the methods according to the described examples include hard disks or optical disks, flash memory, USB devices equipped with non-volatile memory, networked storage devices, etc.

[0111] Devices implementing these disclosed processes and methods can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks can be stored in a computer-readable or machine-readable medium. Multiple processors can perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, standalone devices, etc. The functionality described herein can also be embodied in peripheral devices or add-in cards. As another example, such functionality can also be implemented on a circuit board between different chips or different processes that execute in a single device.

[0112] Instructions, media for transmitting these instructions, computing resources for executing these instructions, and other structures for supporting these computing resources are example components for providing the functionality described in this disclosure.

[0113] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, while illustrative embodiments of this application have been described in detail herein, it should be understood that the concept of this disclosure may be embodied and employed in other ways, and the appended claims are intended to be construed as including these variations, except as limited by the prior art. The various features and aspects of the application described above may be used alone or in combination. Furthermore, without departing from the scope of this specification, the embodiments can be used in any number of environments and applications other than those described herein. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than that described.

[0114] Those skilled in the art will understand that, without departing from the scope of this specification, the less than ("<") and greater than (">") symbols or terms used herein can be replaced by the less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively.

[0115] When a component is described as being “configured” to perform certain operations, this configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0116] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0117] The claim language, or other language, stating "at least one of" and / or "one or more of" in a set indicates that one or more members of the set (in any combination) satisfy the claim. For example, the claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The use of "at least one" and / or "one or more" in a set of languages ​​does not limit the set to items listed in the set. For example, the claim language stating "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.

[0118] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0119] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses (including applications in wireless communication devices, mobile phones, and other devices). Any feature described as a module or component can be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, the techniques can be implemented at least in part by a computer-readable data storage medium comprising program code, which includes instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, wherein the computer program product may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium, wherein the computer-readable communication medium carries or transmits program code in the form of instructions or data structures and is accessible, read and / or executed by a computer, for example, to propagate signals or waves.

[0120] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. This processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0121] The aspects shown in this disclosure include:

[0122] Aspect 1. A method for interrupt handling, comprising: obtaining first image processing configuration data associated with a first image sensor at a first context buffer associated with a first interrupt handler; obtaining second image processing configuration data associated with a second image sensor different from the first image sensor at a second context buffer associated with a second interrupt handler; obtaining an error indication associated with the first image sensor at the first interrupt handler; performing an error handling operation by the first interrupt handler based on the error indication associated with the first image sensor, wherein the error handling operation includes at least one of refreshing the first context buffer or invalidating the first image processing configuration data in the first context buffer; at the second interrupt handler, obtaining an interrupt request IRQ from the second image sensor during a time window interval between obtaining the error indication associated with the first image sensor and the completion time of the error handling operation; and outputting the second image processing configuration data to an image processor based on the obtained IRQ.

[0123] Aspect 2. The method for interrupt handling according to aspect 1, wherein the IRQ corresponds to the completion of an image capture operation performed by a second image sensor.

[0124] Aspect 3. The method for interrupt handling according to aspect 2, wherein the image capture operation includes storing the image in a memory.

[0125] Aspect 4. The method for interrupt handling according to any one of aspects 1 to 3 further includes: obtaining second image processing configuration data at an image processor; and processing an image associated with the second image processing configuration data based on the second image processing configuration data.

[0126] Aspect 5. The method for interrupt handling according to aspect 4 further includes: obtaining third image processing configuration data associated with a third image sensor at a third context buffer associated with a third interrupt handler; obtaining an additional IRQ from the third image sensor by the third interrupt handler; obtaining an indication from the image processor that processing of an image associated with the second image processing configuration data has been completed; and outputting the third image processing configuration data to the image processor by the third interrupt handler based on obtaining the additional IRQ.

[0127] Aspect 6. The method for interrupt handling according to any one of aspects 1 to 5, wherein the plurality of image sensors includes a first image sensor and a second image sensor, and wherein the plurality of interrupt handlers includes a first interrupt handler and a second interrupt handler, wherein the interrupt handlers among the plurality of interrupt handlers are configured to handle IRQs from the plurality of image sensors in a cyclic sequence.

[0128] Aspect 7. The method for interrupt handling according to any one of aspects 1 to 6, wherein the plurality of image sensors includes a first image sensor and a second image sensor, and wherein the plurality of interrupt handlers includes a first interrupt handler and a second interrupt handler, and wherein the interrupt handlers among the plurality of interrupt handlers are configured to handle IRQs from the plurality of image sensors in priority order, wherein a high priority group includes at least one image sensor among the plurality of image sensors, and a low priority group includes at least one different image sensor among the plurality of image sensors.

[0129] Aspect 8. The method for interrupt handling according to any one of aspects 1 to 7, wherein the first image processing configuration data includes at least one of exposure settings, gain, resolution, pixel configuration, motion indication, or dynamic range settings.

[0130] Aspect 9. The method for interrupt handling according to any one of aspects 1 to 8, wherein the first context buffer and the second context buffer are first-in-first-out (FIFO) buffers.

[0131] Aspect 10. The method for interrupt handling according to any one of aspects 1 to 9, wherein the error handling operation includes resetting the first image sensor.

[0132] Aspect 11. The method for interrupt handling according to any one of aspects 1 to 10, wherein the error handling operation includes a non-blocking wait associated with at least one of the second image sensor or the second context buffer.

[0133] Aspect 12. The method for interrupt handling according to aspect 11, wherein, during a non-blocking wait period, the second image sensor remains operable while at least one of the first image sensor or the first context buffer associated with the first interrupt handler is inoperable.

[0134] Aspect 13. An apparatus for interrupt handling, the apparatus comprising: a first interrupt handler configured to obtain an error indication associated with a first image sensor; a first context buffer associated with the first interrupt handler and configured to obtain first image processing configuration data associated with the first image sensor; a second interrupt handler; and a second context buffer associated with the second interrupt handler and configured to obtain second image processing configuration data associated with a second image sensor different from the first image sensor; wherein the first interrupt handler is further configured to perform an error handling operation based on the error indication associated with the first image sensor, wherein the error handling operation includes at least one of refreshing the first context buffer or invalidating the first image processing configuration data in the first context buffer; and wherein the second interrupt handler is configured to obtain an interrupt request IRQ from the second image sensor during a time window interval between obtaining the error indication associated with the first image sensor and the completion time of the error handling operation; and wherein the second context buffer is configured to output the second image processing configuration data to an image processor based on the obtained IRQ.

[0135] Aspect 14. The apparatus according to aspect 13, wherein the IRQ corresponds to the completion of an image capture operation performed by the second image sensor.

[0136] Aspect 15. The apparatus according to aspect 14, wherein the image capture operation includes storing the image in a memory.

[0137] Aspect 16. The apparatus according to any one of aspects 13 to 15 further includes an image processor configured to: obtain second image processing configuration data; and process an image associated with the second image processing configuration data based on the second image processing configuration data.

[0138] Aspect 17. The apparatus according to aspect 16 further includes: a third context buffer associated with a third interrupt handler and configured to obtain third image processing configuration data associated with a third image sensor; and a third interrupt handler, wherein the third interrupt handler is configured to: obtain an additional IRQ from the third image sensor; obtain an indication from the image processor that processing of an image associated with the second image processing configuration data has been completed; and output the third image processing configuration data to the image processor based on obtaining the additional IRQ.

[0139] Aspect 18. The apparatus according to any one of aspects 13 to 17, wherein the plurality of image sensors includes a first image sensor and a second image sensor, and wherein the plurality of interrupt handlers includes a first interrupt handler and a second interrupt handler, wherein the interrupt handlers among the plurality of interrupt handlers are configured to process IRQs from the plurality of image sensors in a cyclic sequence.

[0140] Aspect 19. The apparatus according to any one of aspects 13 to 18, wherein the plurality of image sensors includes a first image sensor and a second image sensor, and wherein the plurality of interrupt handlers includes a first interrupt handler and a second interrupt handler, and wherein the interrupt handlers among the plurality of interrupt handlers are configured to handle IRQs from the plurality of image sensors in priority order, wherein a high priority group includes at least one image sensor among the plurality of image sensors, and a low priority group includes at least one different image sensor among the plurality of image sensors.

[0141] Aspect 20. The apparatus according to any one of aspects 13 to 19, wherein the first image processing configuration data includes at least one of exposure settings, gain, resolution, pixel configuration, motion indication, or dynamic range settings.

[0142] Aspect 21. The apparatus according to any one of aspects 13 to 20, wherein the first context buffer and the second context buffer are first-in-first-out (FIFO) buffers.

[0143] Aspect 22. The apparatus according to any one of aspects 13 to 21, wherein the error handling operation includes resetting the first image sensor.

[0144] Aspect 23. The apparatus according to any one of aspects 13 to 22, wherein the error handling operation includes a non-blocking wait associated with at least one of the second image sensor or the second context buffer.

[0145] Aspect 24. The apparatus according to any one of aspects 13 to 23, wherein, during a non-blocking wait period, the second image sensor remains operable while at least one of the first image sensor or the first context buffer associated with the first interrupt handler is inoperable.

[0146] Aspect 25. A non-transitory computer-readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause at least one processor to perform an operation according to any one of aspects 1 to 12.

[0147] Aspect 26. An apparatus for interrupting operations, comprising one or more components for performing the operations described in any one of aspects 1 to 12.

Claims

1. A method for interrupt handling, comprising: First image processing configuration data associated with the first image sensor is obtained at the first context buffer associated with the first interrupt handler; Second image processing configuration data, which is different from the first image sensor, is obtained at the second context buffer associated with the second interrupt handler; An error indication associated with the first image sensor is obtained at the first interrupt handler; The first interrupt handler performs an error handling operation based on the error indication associated with the first image sensor, wherein the error handling operation includes at least one of refreshing the first context buffer or invalidating the first image processing configuration data in the first context buffer; At the second interrupt handler, an interrupt request (IRQ) is obtained from the second image sensor during a time window interval between obtaining the error indication associated with the first image sensor and the completion time of the error handling operation; and Based on the obtained IRQ, the second image processing configuration data is output to the image processor.

2. The method for handling interruptions according to claim 1, wherein, The IRQ corresponds to the completion of the image capture operation performed by the second image sensor.

3. The method for handling interruptions according to claim 2, wherein, The image capture operation includes storing the image in memory.

4. The method for handling interruptions according to claim 1, further comprising: Obtain the second image processing configuration data at the image processor; as well as The image associated with the second image processing configuration data is processed based on the second image processing configuration data.

5. The method for handling interruptions according to claim 4, further comprising: Obtain the third image processing configuration data associated with the third image sensor at the third context buffer associated with the third interrupt handler; The additional IRQ is obtained from the third image sensor by the third interrupt handler; An indication is obtained from the image processor that the processing of the image associated with the second image processing configuration data has been completed; as well as The third interrupt handler outputs the third image processing configuration data to the image processor based on the obtained additional IRQ.

6. The method for handling interruptions according to claim 1, wherein, The plurality of image sensors include the first image sensor and the second image sensor, and wherein the plurality of interrupt handlers include the first interrupt handler and the second interrupt handler, wherein the interrupt handlers of the plurality of interrupt handlers are configured to process IRQs from the plurality of image sensors in a cyclic sequence.

7. The method for handling interruptions according to claim 1, wherein, The plurality of image sensors include the first image sensor and the second image sensor, and wherein the plurality of interrupt handlers include the first interrupt handler and the second interrupt handler, and wherein the interrupt handlers among the plurality of interrupt handlers are configured to handle IRQs from the plurality of image sensors in priority order, wherein a high priority group includes at least one of the plurality of image sensors, and a low priority group includes at least one different image sensor among the plurality of image sensors.

8. The method for handling interruptions according to claim 1, wherein, The first image processing configuration data includes at least one of exposure settings, gain, resolution, pixel configuration, motion indication, or dynamic range settings.

9. The method for interruption handling according to claim 1, wherein, The first context buffer and the second context buffer are first-in-first-out (FIFO) buffers.

10. The method for handling interruptions according to claim 1, wherein, The error handling operation includes resetting the first image sensor.

11. The method for interrupt handling according to claim 1, wherein, The error handling operation includes a non-blocking wait associated with at least one of the second image sensor or the second context buffer.

12. The interruption handling method according to claim 11, wherein, During the non-blocking wait period, the second image sensor remains operational, while at least one of the first image sensor or the first context buffer associated with the first interrupt handler is inoperable.

13. An apparatus for interrupt handling, the apparatus comprising: The first interrupt handler is configured to receive an error indication associated with the first image sensor; A first context buffer, associated with the first interrupt handler, is configured to acquire first image processing configuration data associated with the first image sensor; Second interrupt handler; as well as A second context buffer, associated with the second interrupt handler, is configured to obtain second image processing configuration data associated with a second image sensor that is different from the first image sensor; The first interrupt handler is further configured to perform an error handling operation based on obtaining the error indication associated with the first image sensor, wherein the error handling operation includes at least one of refreshing the first context buffer or invalidating the first image processing configuration data in the first context buffer; and The second interrupt handler is configured to obtain an interrupt request IRQ from the second image sensor during a time window interval between obtaining the error indication associated with the first image sensor and the completion time of the error handling operation; and The second context buffer is configured to output the second image processing configuration data to the image processor based on the obtained IRQ.

14. The apparatus according to claim 13, wherein, The IRQ corresponds to the completion of the image capture operation performed by the second image sensor.

15. The apparatus according to claim 14, wherein, The image capture operation includes storing the image in memory.

16. The apparatus of claim 13, further comprising an image processor configured to: Obtain the second image processing configuration data; and The image associated with the second image processing configuration data is processed based on the second image processing configuration data.

17. The apparatus of claim 16, further comprising: The third context buffer, associated with the third interrupt handler, is configured to acquire third image processing configuration data associated with the third image sensor; as well as The third interrupt handler is configured to: Additional IRQ is obtained from the third image sensor; An indication is obtained from the image processor that the processing of the image associated with the second image processing configuration data has been completed; as well as Based on the obtained additional IRQ, the third image processing configuration data is output to the image processor.

18. The device according to claim 13, wherein, The plurality of image sensors include the first image sensor and the second image sensor, and wherein the plurality of interrupt handlers include the first interrupt handler and the second interrupt handler, wherein the interrupt handlers of the plurality of interrupt handlers are configured to process IRQs from the plurality of image sensors in a cyclic sequence.

19. The apparatus according to claim 13, wherein, The plurality of image sensors include the first image sensor and the second image sensor, and wherein the plurality of interrupt handlers include the first interrupt handler and the second interrupt handler, and wherein the interrupt handlers among the plurality of interrupt handlers are configured to handle IRQs from the plurality of image sensors in priority order, wherein a high priority group includes at least one of the plurality of image sensors, and a low priority group includes at least one different image sensor among the plurality of image sensors.

20. The apparatus according to claim 13, wherein, The first image processing configuration data includes at least one of exposure settings, gain, resolution, pixel configuration, motion indication, or dynamic range settings.

21. The apparatus according to claim 13, wherein, The first context buffer and the second context buffer are first-in-first-out (FIFO) buffers.

22. The apparatus according to claim 13, wherein, The error handling operation includes resetting the first image sensor.

23. The apparatus according to claim 13, wherein, The error handling operation includes a non-blocking wait associated with at least one of the second image sensor or the second context buffer.

24. The apparatus according to claim 23, wherein, During the non-blocking wait period, the second image sensor remains operational, while at least one of the first image sensor or the first context buffer associated with the first interrupt handler is inoperable.

Images