Method and system for shift estimation of one or more output frames
By receiving sensor data from the image capture device, using deep learning neural network to predict alignment shifts between frames, the problem of irregularity caused by inter-frame delay in image capture device is solved, and the generation of high dynamic range images is realized, which improves the Image quality.
Patent Information
- Application Number
- CN202380035761.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-02
- Filing Date
- 2023-03-20
- Publication Date
- 2024-12-06
AI Technical Summary
In an image capturing device, the delay between long exposure frames and short exposure frames leads to image irregularity, and the prior art is difficult to effectively align, affecting the generation of high dynamic range images.
By receiving sensor data from the image capture device, using a deep learning neural network based on machine learning to predict alignment shifts between frames and applying this shift when the frame is captured, aligning features of long-exposure frames and short-exposure frames, thereby Generate high dynamic range images.
Effective alignment of long-exposure frames and short-exposure frames is achieved, image irregularity is reduced, the quality of high-dynamic range images is improved, and the performance of image capture devices is enhanced.
Smart Images

Figure CN119096267A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to exposure control for image capture devices. In some examples, aspects of the present disclosure relate to aligning frames (or images) associated with different exposure times based on an estimated shift of the frames (or images). Background Art
[0002] A camera is a device that uses an image sensor to receive light and capture image frames, such as still images or video frames. A camera may include a processor, such as an image signal processor (ISP), that may receive one or more image frames and process one or more image frames. For example, raw image frames captured by a camera sensor may be processed by an ISP to generate a final image. A camera may be configured with various image capture and image processing settings to change the appearance of an image. Some camera settings, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus, and gain, are determined and applied before or during the capture of a photo. Other camera settings may configure post-processing of a photo, such as changes in contrast, brightness, saturation, sharpness, levels, curves, or color.
[0003] The camera can be configured with various image capture and image processing settings. Applying different settings can produce frames or images with different appearances. Some camera settings, such as ISO, exposure time (also called exposure duration), aperture size, f / stop, shutter speed, focus, and gain, are determined and applied before or during the capture of a photo. Other camera settings can configure post-processing of the photo, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color.
[0004] In photography, the exposure of an image captured by a camera refers to the amount of light per unit area that reaches the photographic film or, in modern cameras, the electronic image sensor. The exposure is based on camera settings, such as shutter speed, exposure time, and / or lens aperture, and the brightness of the scene being photographed. Many cameras are equipped with an automatic exposure or "autoexposure" mode, in which the exposure settings (e.g., shutter speed, exposure time, lens aperture, etc.) may be automatically adjusted to match as closely as possible the brightness of the scene or object being photographed. In some cases, an automatic exposure control (AEC) engine may perform AEC to determine the exposure setting of the image sensor. Summary of the invention
[0005] In some examples, systems and techniques of xxx are described. According to at least one illustrative example, a method for capturing an image is provided. The method includes: xxx.
[0006] Certain aspects provide an apparatus for frame alignment. The apparatus generally includes: a memory; and at least one processor coupled to the memory and configured to: receive sensor data indicating movement associated with an image capture device that captures a first frame and a second frame, wherein the first frame is associated with a higher exposure setting than the second frame; predict a shift for aligning one or more features of the first frame with one or more features of the second frame based on the sensor data; apply the shift to align the one or more features of the first frame with the one or more features of the second frame; and generate a frame output at least in part by combining at least a portion of the first frame with at least a portion of the second frame.
[0007] Certain aspects provide a method for frame alignment. The method generally includes: receiving sensor data indicating movement associated with an image capture device that captures a first frame and a second frame, wherein the first frame is associated with a higher exposure setting than the second frame; predicting a shift for aligning one or more features of the first frame with one or more features of the second frame based on the sensor data; applying the shift to align the one or more features of the first frame with the one or more features of the second frame; and generating a frame output at least in part by combining at least a portion of the first frame with at least a portion of the second frame.
[0008] Certain aspects provide a non-transitory computer-readable medium having instructions that, when executed by one or more processors, cause the one or more processors to: receive sensor data indicating movement associated with an image capture device that captures a first frame and a second frame, wherein the first frame is associated with a higher exposure setting than the second frame; predict, based on the sensor data, a shift for aligning one or more features of the first frame with one or more features of the second frame; apply the shift to align the one or more features of the first frame with the one or more features of the second frame; and generate a frame output at least in part by combining at least a portion of the first frame with at least a portion of the second frame.
[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.
[0010] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:
[0012] Figure 1A , Figure 1B and Figure 1C is a diagram illustrating example configurations of image sensors of image capture devices according to some examples.
[0013] Figure 2 is a block diagram illustrating the architecture of an image capture and processing device.
[0014] Figure 3 is a block diagram illustrating an example of an image capture system.
[0015] Figure 4 Techniques for generating a fused frame from a short exposure frame and a long exposure frame are illustrated.
[0016] Figure 5 A long exposure stream and a short exposure stream from an image sensor are illustrated.
[0017] Figure 6 An example image processing system is illustrated.
[0018] Figure 7 Techniques for online fusion of short and long frames in accordance with certain aspects of the present disclosure are illustrated.
[0019] Figure 8 is a block diagram illustrating example techniques for frame alignment in accordance with certain aspects of the present disclosure.
[0020] Fig. 9 An example image processing system using alignment prediction in accordance with certain aspects of the present disclosure is illustrated.
[0021] Fig.10 is a flowchart illustrating an example of a process of capturing an image.
[0022] Fig.11 is an illustrative example of a deep learning neural network that can be used to implement machine learning based alignment prediction.
[0023] Fig.12 is an illustrative example of a convolutional neural network (CNN).
[0024] Fig.13 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION
[0025] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.
[0026] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of the present application as set forth in the appended claims.
[0027] Described herein are systems, devices, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for aligning frames (or images or image frames) with different exposure times. As used herein, the terms image, frame, and image frame are used interchangeably. In order to generate a high dynamic range (HDR) image (e.g., frame), a long exposure frame and a short exposure frame may be captured using the same image sensor and fused to generate an HDR image. Therefore, there may be a delay between when the long exposure frame is captured and when the short exposure frame is captured. Any movement during this delay may cause irregularities in the image or the final video being generated. Therefore, the pixels of the long exposure frame (or short exposure frame) may be shifted to align the features of the long exposure frame (or short exposure frame) with the features of the short exposure frame (or long exposure frame).
[0028] The fusion of long exposure frames with short exposure frames can be done online (e.g., in real time as each frame is captured), which is more efficient than performing the fusion once the entire frame has been captured. However, with such online operations, alignment may have to be programmed several frames in advance. Certain aspects provide techniques for estimating shifts for alignment of frames based on past samples. For example, a neural network can be used to estimate shifts for alignment in the x and y dimensions (e.g., direction) based on previous samples for online fusion. Once the shift of a particular frame is estimated in advance, the shift can be applied for alignment once the frame is captured. As used herein, long exposure frames and short exposure frames generally refer to any frames with different exposures, where the long exposure frame has a longer exposure than the short exposure frame. In addition, although some examples provided herein describe techniques for fusing frames with different exposure times, aspects of the present disclosure may be applied to fusing frames with different exposure settings (e.g., fusion of high exposure frames with low exposure frames).
[0029] Image sensors include one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor. In some cases, different photodiodes may be covered by different color filters of a color filter array and may therefore measure light that matches the color of the color filter covering the photodiode.
[0030] Various color filter arrays may be used, including a Bayer color filter array, a four-color color filter array (also referred to as a four-color Bayer filter or QCFA), and / or other color filter arrays. Figure 1A An example of a Bayer color filter array 100 is shown in FIG. As shown, the Bayer color filter array 100 includes a repeating pattern of red, blue, and green filters. Figure 1B As shown, QCFA 110 includes a 2×2 (or "four-color") pattern of color filters, which includes a 2×2 pattern of red (R) color filters, a pair of 2×2 patterns of green (G) color filters, and a 2×2 pattern of blue (B) color filters. For the entire array of photodiodes of a given image sensor, the Figure 1B 10. Whether using QCFA 110 or Bayer color filter array 100, each pixel of the image is generated based on the following data: red light data from at least one photodiode covered by the red filter of the color filter array, blue light data from at least one photodiode covered by the blue filter of the color filter array, and green light data from at least one photodiode covered by the green filter of the color filter array. Other types of color filter arrays may use yellow, magenta and / or cyan (also called "emerald") filters instead of or in addition to the red, blue and / or green filters. Different photodiodes throughout the pixel array may have different spectral sensitivity curves and therefore respond to different wavelengths of light. Monochrome image sensors may also lack color filters and, therefore, lack color depth.
[0031] In some cases, a subset of adjacent photodiodes (e.g., when using Figure 1B When the QCFA 110 is shown, a 2×2 patch of photodiodes can measure light of the same color from approximately the same area of the scene. For example, when the photodiodes included in each subset of photodiodes are in close physical proximity, the light incident on each photodiode of the subset can originate from approximately the same location in the scene (e.g., a portion of leaves on a tree, a small portion of the sky, etc.).
[0032] In some examples, the brightness range of light from a scene can significantly exceed the brightness levels that an image sensor can capture. For example, a digital single-lens reflex (DSLR) camera may be able to capture a 1:30,000 contrast ratio of light from a scene, while the brightness levels of an HDR scene can exceed a 1:1,000,000 contrast ratio.
[0033] In some cases, an HDR sensor may be utilized to enhance the contrast ratio of an image captured by an image capture device. In some examples, an HDR sensor may be used to obtain multiple exposures within an image or frame, where such multiple exposures may include a short exposure time (e.g., 5 ms) and a long exposure time (e.g., 15 ms or longer). As used herein, a long exposure time generally refers to any exposure time that is longer than a short exposure time.
[0034] In some implementations, an HDR sensor may be able to configure individual photodiodes within a subgroup of photodiodes (e.g., from Figure 1B The four individual R photodiodes, four individual B photodiodes, and four individual G photodiodes in each of the two 2×2 G patches in QCFA 110 are shown with different exposure settings. A collection of photodiodes with matching exposure settings is also referred to herein as a photodiode exposure group. Figure 1C A portion of an image sensor array with a QCFA filter configured with four different photodiode exposure groups 1 to 4 is illustrated. Figure 1C As shown in the example photodiode exposure group array 120 in FIG. 1 , each 2×2 tile may include photodiodes from each photodiode exposure group in a different photodiode exposure group for a particular image sensor. Figure 1C Four groups are shown in the specific grouping in FIG. 1 , but one of ordinary skill will recognize that a different number of photodiode exposure groups, a different arrangement of photodiode exposure groups within a subgroup, and any combination thereof may be used without departing from the scope of the present disclosure.
[0035] As compared to Figure 1CAs noted, in some HDR image sensor implementations, the exposure settings corresponding to different photodiode exposure groups may include different exposure times (also referred to as exposure durations), such as short exposure, medium exposure, and long exposure. In some cases, different images of a scene associated with different exposure settings may be formed by light captured by the photodiodes of each photodiode exposure group. For example, a first image may be formed by light captured by the photodiodes of photodiode exposure group 1, a second image may be formed by photodiodes of photodiode exposure group 2, a third image may be formed by light captured by photodiodes of photodiode exposure group 3, and a fourth image may be formed by light captured by photodiodes of photodiode exposure group 4. Based on the differences in the exposure settings corresponding to each group, the brightness of objects in the scene captured by the image sensor may be different in each image. For example, a well-lit object captured by a photodiode with a long exposure setting may appear saturated (e.g., completely white). In some cases, the image processor may select between pixels of images corresponding to different exposure settings to form a combined image.
[0036] In one illustrative example, the first image corresponds to a short exposure time (also referred to as a short exposure image), the second image corresponds to a medium exposure time (also referred to as a medium exposure image), and the third image and the fourth image correspond to long exposure times (also referred to as long exposure images). In such an example, pixels of the combined image corresponding to a portion of the scene with low illumination (e.g., a portion of the scene in shadow) may be selected from a long exposure image (e.g., the third image or the fourth image). Similarly, pixels of the combined image corresponding to a portion of the scene with high illumination (e.g., a portion of the scene in direct sunlight) may be selected from a short exposure image (e.g., the first image).
[0037] In some cases, the image sensor may also utilize a photodiode exposure group to capture an object in motion without blurring. The length of the exposure time of the photodiode group may correspond to the distance that an object in the scene moves during the exposure time. If light from an object in motion is captured by a photodiode corresponding to multiple image pixels during the exposure time, the object in motion may appear blurred across multiple image pixels (also known as motion blur). In some implementations, motion blur may be reduced by configuring one or more photodiode groups with a short exposure time. In some implementations, an image capture device (e.g., a camera) may determine the amount of local motion (e.g., a motion gradient) within a scene by comparing the position of an object between two consecutive captured images. For example, motion may be detected in a preview image captured by an image capture device to provide a preview function to a user on a display. In some cases, a machine learning model may be trained to detect local motion between consecutive images.
[0038] Various aspects of the technology described herein will be discussed below with respect to the various figures. Figure 2 2 is a block diagram illustrating the architecture of an image capture and processing system 200. The image capture and processing system 200 includes various components for capturing and processing an image of a scene (e.g., an image of the scene 210). The image capture and processing system 200 may capture an independent image (or photograph) and / or may capture a video including multiple images (or video frames) in a particular sequence. In some cases, the lens 215 and the image sensor 230 may be associated with an optical axis. In an illustrative example, the photosensitive area of the image sensor 230 (e.g., a photodiode) and the lens 215 may be centered on the optical axis. The lens 215 of the image capture and processing system 200 faces the scene 210 and receives light from the scene 210. The lens 215 bends the incoming light from the scene toward the image sensor 230. The light received by the lens 215 passes through an aperture. In some cases, the aperture (e.g., the aperture size) is controlled by one or more control mechanisms 220 and received by the image sensor 230. In some cases, the aperture may have a fixed size.
[0039] The one or more control mechanisms 220 may control exposure, focus, and / or zoom based on information from the image sensor 230 and / or based on information from the image processor 250. The one or more control mechanisms 220 may include multiple mechanisms and components; for example, the control mechanisms 220 may include one or more exposure control mechanisms 225A, one or more focus control mechanisms 225B, and / or one or more zoom control mechanisms 225C. The one or more control mechanisms 220 may also include additional control mechanisms other than those illustrated, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.
[0040] A focus control mechanism 225B of the control mechanism 220 may obtain a focus setting. In some examples, the focus control mechanism 225B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 225B may adjust the positioning of the lens 215 relative to the positioning of the image sensor 230. For example, based on the focus setting, the focus control mechanism 225B may adjust the focus by actuating a motor or servo system (or other lens mechanism) to move the lens 215 closer to the image sensor 230 or further away from the image sensor 230. In some cases, an additional lens may be included in the image capture and processing system 200, such as one or more micro lenses located above each photodiode of the image sensor 230, each of which bends the light received from the lens 215 toward the corresponding photodiode before the light reaches the corresponding photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting may be determined using the control mechanism 220, the image sensor 230, and / or the image processor 250. The focus setting may be referred to as an image capture setting and / or an image processing setting. In some cases, lens 215 may be fixed relative to the image sensor, and focus control mechanism 225B may be omitted without departing from the scope of the present disclosure.
[0041] The exposure control mechanism 225A of the control mechanism 220 may obtain an exposure setting. In some cases, the exposure control mechanism 225A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control mechanism 225A may control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the duration that the sensor collects light (e.g., exposure time or electronic shutter speed), the sensitivity of the image sensor 230 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 230, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.
[0042] The zoom control mechanism 225C of the control mechanism 220 may obtain a zoom setting. In some examples, the zoom control mechanism 225C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 225C may control the focal length of a lens element assembly (lens assembly) including the lens 215 and one or more additional lenses. For example, the zoom control mechanism 225C may control the focal length of the lens assembly by actuating one or more motors or servos (or other lens mechanisms) to move one or more lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (which may be the lens 215 in some cases) that first receives light from the scene 210, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 215) and the image sensor 230 before the light reaches the image sensor 230. In some cases, the afocal zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference between each other), and a negative (e.g., diverging, concave) lens between the two positive lenses. In some cases, zoom control mechanism 225C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lens. In some cases, zoom control mechanism 225C may control the zoom by capturing images from an image sensor (e.g., including image sensor 230) of a plurality of image sensors at a zoom corresponding to the zoom setting. For example, image processing system 200 may include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a greater zoom. In some cases, based on the selected zoom setting, zoom control mechanism 225C may capture images from the corresponding sensors.
[0043] Image sensor 230 includes one or more arrays of photodiodes or other light sensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image produced by image sensor 230. In some cases, different photodiodes may be covered by different filters. In some cases, different photodiodes may be covered in different color filters and may therefore measure light that matches the color of the filter covering the photodiode. Various color filter arrays may be used, including Bayer color filter arrays (e.g., Figure 1A shown), QCFA (see Figure 1B ) and / or any other color filter array.
[0044] return Figure 1A and Figure 1B, other types of color filters may use yellow filters, magenta filters, and / or cyan (also known as "emerald") filters instead of or in addition to red filters, blue filters, and / or green filters. In some cases, some photodiodes may be configured to measure infrared (IR) light. In some implementations, the photodiode measuring IR light may not be covered by any filter, thereby allowing the IR photodiode to measure both visible light (e.g., color) and IR light. In some examples, the IR photodiode may be covered by an IR filter, thereby allowing IR light to pass through and blocking light from other parts of the spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 230) may lack filters entirely (e.g., color, IR, or any other part of the spectrum), and may instead use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, and therefore respond to light of different wavelengths. Monochrome image sensors may also lack filters, and therefore lack color depth.
[0045] In some cases, image sensor 230 may alternatively or additionally include opaque and / or reflective masks that block light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, opaque and / or reflective masks may be used for phase detection autofocus (PDAF). In some cases, opaque and / or reflective masks may be used to block portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., IR cutoff filters, UV cutoff filters, bandpass filters, low pass filters, high pass filters, etc.). Image sensor 230 may also include an analog gain amplifier for amplifying an analog signal output by a photodiode, and / or an analog-to-digital converter (ADC) for converting an analog signal output by a photodiode (and / or an analog signal amplified by an analog gain amplifier) into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms in control mechanism 220 may be alternatively or additionally included in image sensor 230. Image sensor 230 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.
[0046] Image processor 250 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 254), one or more host processors (including host processor 252), and / or Fig.13The host processor 252 may be a digital signal processor (DSP) and / or other type of processor. In some implementations, the image processor 250 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 252 and the ISP 254. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 256), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM , global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 256 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 252 may communicate with the image sensor 230 using an I2C port, and the ISP 254 may communicate with the image sensor 230 using a MIPI port.
[0047] The image processor 250 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 250 may store image frames and / or processed images in a random access memory (RAM) 240 / 925, a read-only memory (ROM) 245 / 920, a cache, a memory unit, another storage device, or some combination thereof.
[0048] Various input / output (I / O) devices 260 may be connected to the image processor 250. The I / O devices 260 may include a display screen, a keyboard, a keypad, a touch screen, a touch pad, a touch-sensitive surface, a printer, any other output device 1335, any other input device 1345, or some combination thereof. In some cases, subtitles may be entered into the image processing device 205B via a physical keyboard or keypad of the I / O device 260, or via a virtual keyboard or keypad of a touch screen of the I / O device 260. The I / O 260 may include one or more ports, jacks, or other connectors that enable a wired connection between the image capture and processing system 200 and one or more peripheral devices, through which the image capture and processing system 200 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The I / O 260 may include one or more wireless transceivers that enable a wireless connection between the image capture and processing system 200 and one or more peripheral devices, through which the image capture and processing system 200 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 260 discussed previously, and may themselves be considered I / O devices 260 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.
[0049] In some cases, the image capture and processing system 200 can be a single device. In some cases, the image capture and processing system 200 can be two or more independent devices, including an image capture device 205A (e.g., a camera) and an image processing device 205B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 205A and the image processing device 205B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 205A and the image processing device 205B can be disconnected from each other.
[0050] like Figure 2 As shown, the vertical dotted line will Figure 2 The image capture and processing system 200 is divided into two parts, representing an image capture device 205A and an image processing device 205B. The image capture device 205A includes a lens 215, a control mechanism 220, and an image sensor 230. The image processing device 205B includes an image processor 250 (including an ISP 254 and a host processor 252), a RAM 240, a ROM 245, and an I / O 260. In some cases, some components illustrated in the image capture device 205A (such as the ISP 254 and / or the host processor 252) may be included in the image capture device 205A.
[0051] The image capture and processing system 200 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 200 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some specific implementations, the image capture device 205A and the image processing device 205B may be different devices. For example, the image capture device 205A may include a camera device, and the image processing device 205B may include a computing device, such as a mobile phone, a desktop computer, or other computing devices.
[0052] Although the image capture and processing system 200 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 200 may include more than Figure 2 Those components shown are more components. The components of the image capture and processing system 200 may include software, hardware, or one or more combinations of software and hardware. For example, in some specific implementations, the components of the image capture and processing system 200 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 200.
[0053] Figure 3 is a block diagram illustrating an example of an image capture system 300. The image capture system 300 includes various components for processing input images or frames to produce output images or frames. As shown, the components of the image capture system 300 include one or more image capture devices 302, an image processing engine 310, and an output device 312. The image processing engine 310 can produce a high dynamic range depiction of a scene, as described in more detail herein.
[0054] The image capture system 300 may include or be part of an electronic device or system. For example, the image capture system 300 may include or be part of an electronic device or system, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle or a computing device / system of a vehicle, a server computer (e.g., communicating with another device or system (such as a mobile device, an XR system / device, a vehicle computing system / device, etc.)), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera device, a display device, a digital media player, a video streaming device, or any other suitable electronic device. In some examples, the image capture system 300 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communications (such as cellular network communications, 802.11 Wi-Fi communications, wireless local area network (WLAN) communications, Bluetooth or other short-range communications, any combination thereof, and / or other communications). In some implementations, the components of image capture system 300 can be part of the same computing device. In some implementations, the components of image capture system 300 can be part of two or more separate computing devices.
[0055] Although image capture system 300 is shown as including certain components, one of ordinary skill will appreciate that exposure control system 300 may include more than Figure 3 In some cases, the additional components of the image capture system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, the exposure control system 300 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radars, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or Figure 3One or more other software and / or hardware components not shown in the figure. In some specific implementations, the additional components of the exposure control system 300 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware (these electronic circuits or other electronic hardware may include one or more programmable electronic circuits (e.g., digital signal processors (DSPs), microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), any combination thereof, and / or other suitable electronic circuits)), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the exposure control system 300.
[0056] One or more image capture devices 302 may capture image data and generate an image (or frame) based on the image data, and / or may provide the image data to an image processing engine 310 for further processing. One or more image capture devices 302 may also provide the image data to an output device 312 for output (e.g., on a display). In some cases, the output device 312 may also include a storage device. An image or frame may include an array of pixels representing a scene. For example, the image may be: a red-green-blue (RGB) image having red, green, and blue color components per pixel; a luminance, redness, blueness (YCbCr) image having a luminance component and two chrominance (color) components (redness and blueness) per pixel; or any other suitable type of color or monochrome image. In addition to the image data, the image capture device may also generate supplemental information, such as the amount of time between consecutively captured images, a timestamp of image capture, and the like.
[0057] Figure 4Techniques for generating a fused frame from a short exposure frame and a long exposure frame are illustrated. As shown, a short exposure frame 402 and a long exposure frame 404 may be acquired, which may be fused to provide a fused frame output 406 (e.g., a high dynamic range (HDR) output). Due to the bit depth of the image capture sensor, some pixels of the captured frame may be oversaturated, resulting in the image not showing some textures of the scene as shown in the short exposure frame 402. Therefore, in order to generate an HDR frame, both a short exposure frame and a long exposure frame may be captured, which may be fused (e.g., combined) to generate an HDR output frame. The fusion of the short exposure frame and the long exposure frame may be performed to generate a fused output frame including a portion of the short exposure frame and a portion of the long exposure frame. For example, a region 408 of the fused frame output 406 may be from the long exposure frame 404, and a region 410 of the fused frame output 406 may be from the short exposure frame 402. However, fusing the short exposure frame and the long exposure frame may result in irregularities due to global motion (e.g., motion of the image capture device). For example, from the time the long exposure frame is captured to the time the short exposure frame is captured, the image capture device or objects in the scene may have moved, causing irregularities if steps are not taken to align the short and long exposure frames before the frames are fused together. This global motion problem can also occur due to rolling shutter, as described in more detail herein.
[0058] Figure 5 1 illustrates a long exposure flow and a short exposure flow (e.g., a Mobile Industry Processor Interface (MIPI) flow) from an image sensor (e.g., image sensor 230) to an imaging front end for processing. Line 502 represents the start of long exposure sensing (also referred to herein as normal exposure sensing), and line 504 represents the end of long exposure sensing. As shown, long exposure sensing starts from a sensor (e.g., Figure 2 230) to the last row of the sensor. For each row (e.g., a row of photodiodes), once long exposure sensing has been completed, short exposure sensing begins while long exposure sensing continues to the next row. For example, line 506 represents the beginning of short exposure sensing and line 508 represents the end of short exposure sensing starting from the first row of the image sensor to the last row. Long exposure sensing (e.g., in Figure 5 with a duration marked as “Nnormal”) can be used for short exposure sensing (e.g., Figure 5 with a duration marked as “N short”).
[0059] Once long exposure sensing for a particular row is complete, a short delay occurs (e.g., associated with the gap between lines 504, 506) before short exposure sensing begins. Once short exposure sensing for a particular row has been completed, the information for that row is read out from the image sensor for processing. Due to the gap from long exposure sensing to short exposure sensing (e.g., between Figure 5There is a chance of movement for the user holding the camera and / or for objects in the scene being captured to move, resulting in misalignment of features in the short exposure frame and the long exposure frame (e.g., features that are common or identical in the short exposure frame and the long exposure frame). For example, there may be a motion delay (D) from time 550 (e.g., a time when half of the long exposure data is captured) to time 552 (e.g., a time when half of the short exposure data is captured). The motion delay (D) may be estimated as the average motion delay associated with different long frame and short frame capture events (e.g., different HDR frame captures).
[0060] Since sensing occurs one row at a time (e.g., starting from the first row to the last row), rolling shutter global motion also occurs. The camera or object in the scene may move from when the first row of data of the sensor is captured to when the last row of data of the sensor is captured. Therefore, when reading out the short exposure data and the long exposure data from the sensor, a rolling shutter correction (RSC) process and a frame alignment process may be performed.
[0061] Figure 6 An example image processing system is illustrated. As shown, short exposure frames 602 and long exposure frames 604 may each be input to a respective one of RSC elements 606, 610. In addition, sensor (e.g., gyroscope, accelerometer, IMU, etc.) data and timing data associated with the short exposure frames and the long exposure frames may be input to an electronic image stabilization element 608, which may generate information to be used for rolling shutter correction of the long and short frames. For example, the sensor (e.g., gyroscope, accelerometer, IMU, etc.) data and timing data may be used to estimate the motion associated with the frame (e.g., due to changes in roll, pitch, and yaw of the camera) from when the first row of data is captured by the sensor to when the last row of data is captured by the sensor, which may be used to correct for such motion using the RSC element.
[0062] The output of the RSC element includes a short' frame 612 and a long' frame 616. The short' frame 612 and the long' frame 616 may be provided to an alignment estimation element 614, which may determine the motion associated with the delay from capturing the long and short frames, such as relative to Figure 5 This information may be used to align the short' frame 612 with the long' frame 616 at block 618. Based on the alignment, the short' frame 612 may be adjusted (e.g., by warping pixel data) to generate a short' frame 620. The short' frame 620 and the long' frame 616 may be provided to a fusion engine 622, which may generate an HDR output 624 (e.g., an output fused image, such as with respect to Figure 4 described above).
[0063] Figure 7704 and 702 to generate an HDR frame. Figure 5 As described, long exposure data corresponding to one or more long exposure frames 702 may be captured for each row before short exposure data corresponding to one or more short exposure frames 704. Thus, data for one or more long exposure frames 702 from each row may be received and stored in buffer 703 before data for one or more short exposure frames 704 for each row is stored in buffer 705. As shown, accumulation of data for one or more long exposure frames 702 may precede accumulation of data for one or more short exposure frames 704 (e.g., because long exposure capture occurs before short exposure capture, as shown in FIG. 1 ). Figure 5 shown).
[0064] In some cases, fusion by the fusion engine 706 may begin once short frame data for a particular number of sensor rows or lines (e.g., the first 3 rows / lines, the first 4 rows / lines, the first 8 rows / lines, or another number of rows / lines) corresponding to one or more short exposure frames 704 has been accumulated. For example, operations for frame alignment may begin after short frame data for a particular number of sensor rows has been received (e.g., rather than waiting to receive the entire frame). However, various constraints may exist when performing frame alignment. For example, it may not be possible to fully warp a long exposure frame (from one or more long exposure frames 702) to align with a short exposure frame (from one or more short exposure frames 704). In addition, programming of alignment may have to be performed two or three frames in advance due to hardware timing constraints. In some aspects, a large buffer may be established for capturing frame data. Image data from an image sensor may be written at a central portion of an image buffer, enabling shifts in the x and y dimensions to be applied to data stored in the buffer for alignment. In addition, certain aspects of the present disclosure provide techniques for alignment prediction to allow alignment operations to be programmed in advance.
[0065] Figure 8 8 is a block diagram illustrating an example technique for frame alignment according to certain aspects of the present disclosure. A sensor 802 (e.g., image sensor 230) may generate a long exposure frame 808 and a short exposure frame 810. The short exposure frame 810 may be misaligned with the long exposure frame 808 due to delays associated with capturing and fusing the long and short exposure frames, as described above. The long exposure frame 808 and the short exposure frame 810 are provided to a sensor front end (SFE) 806 for alignment and fusion.
[0066] Sensor data (e.g., gyroscope data, accelerometer data, IMU data, any combination thereof, and / or other data) may be provided to a sensor machine learning (ML) system library 804. In one illustrative example, the sensor ML system library 804 may include a recurrent neural network (RNN) library 804. Using the sensor ML system library 804 (e.g., RNN library), the predicted alignment parameters (e.g., average alignment) may be correlated with the delay between the capture of the short frame and the long frame (e.g., Figure 5 The estimated or average motion delay (D) shown is selected correspondingly and based on the sensor data (e.g., gyroscope data, accelerometer data, IMU data, etc.), as shown. The sensor ML system library 804 (e.g., RNN library) may include an estimate of the alignment offset as a function of the sensor data (e.g., gyroscope data), assuming an average delay between the capture of the short frame sensor data and the long frame sensor data. Therefore, the sensor ML system library 804 (e.g., RNN library) can be used to estimate the pixel shift two or three frames in advance, thereby allowing operations for alignment of future frames to be programmed. Therefore, when a frame is captured and received, the associated shift to be applied to the frame has been estimated, allowing the frame to be shifted. The sensor ML system library 804 (e.g., RNN library) uses historical data to predict frame shifts that can be applied to future frames (e.g., two or three frames in advance).
[0067] The RNN library may output a change in the x dimension (dx) and a change in the y dimension to be applied to the long frame to align with the short frame. The pixels of the long frame may be adjusted accordingly (e.g., shifted based on dx and dy). For example, the SFE 806 may receive parameters dx and dy and perform an online fusion of the short frame and the long frame (e.g., sensor Bayer online fusion) by performing an alignment of the long frame to the short frame using the dx and dy parameters. The image front end (IFE) 812 converts the output from the SFE 806 (e.g., Bayer format output) to a red, green, and blue (RGB) output.
[0068] Fig. 9 An example image processing system using alignment prediction according to certain aspects of the present disclosure is illustrated. As shown, an alignment prediction library 902 (e.g., including Figure 8The sensor ML system library 804) may receive sensor inputs (e.g., gyroscope sensor data, accelerometer data, IMU data, etc.) and timing inputs. The timing input may include a delay (D) between long frame capture and short frame capture. The alignment prediction library 902 may generate an alignment grid (e.g., a 35×27 grid or a grid of another size) that provides a mapping of pixels to align the current long frame to the current short frame. Based on the alignment grid, an average shift of pixels in the x dimension and an average shift of pixels in the y dimension may be determined for the current frame (e.g., by the alignment prediction library 902 or the shift engine 904). The average shifts in the x and y dimensions are input to the sensor ML system library 804 (e.g., RNN library). The sensor ML system library 804 (e.g., RNN library) may also have received the average shifts in the x and y dimensions for one or more previous frames (e.g., frames captured before the current frame). Based on the average shift information of the current frame and the previous frame, the sensor ML system library 804 (e.g., RNN library) predicts the shift of the future frame (e.g., two or three frames in advance). For example, the sensor ML system library 804 (e.g., RNN library) may provide a predicted shift to be applied three frames in advance. The sensor ML system library 804 (e.g., RNN library) may be generated using a machine learning model trained using supervised training techniques. For example, frame data for training the machine learning model may be obtained by implementing various camera movements when capturing images or frames for training.
[0069] Once the shift has been predicted using the RNN library, the shift engine 904 may use the shift prediction to shift the long frame 604 to generate a shifted long frame 906 (labeled as Shift XY Long'). The shifted long frame 906 and the short frame 602 may be provided from the shift engine 904 to the fusion engine 622 to generate an HDR output 624. For example, the fusion engine 622 may determine for each pixel of the short exposure frame whether the saturation associated with the pixel is less than a lower threshold, and if so, use the corresponding pixel of the long exposure frame for the HDR output 624. If the saturation associated with the pixel is greater than an upper threshold, the fusion engine 622 may use the corresponding pixel of the short exposure frame. If the saturation is between the lower threshold and the upper threshold, the corresponding pixels from the short exposure frame and the long exposure frame may be blended to generate the associated pixels for the HDR output. As shown, at block 914, electronic image stabilization (EIS) may be applied to generate a video output (e.g., using the current and future frames to stabilize the video).
[0070] Fig.101 is a flow chart illustrating an example of a process 1000 for capturing an image. The process 1000 may be performed by a frame alignment system, such as the processor 1310 and, in some aspects, the storage device 1330. At block 1002, the process 1000 includes receiving sensor data (e.g., gyroscope data and / or other sensor data) indicating movement associated with an image capture device (e.g., the image capture device 205A) that captured a first frame and a second frame, wherein the first frame is associated with a higher exposure setting (e.g., a longer exposure time) than the second frame.
[0071] At block 1004, the frame alignment system predicts a shift for aligning one or more features of the first frame with one or more features of the second frame based on the sensor data. In some aspects, the shift is predicted before the first frame and the second frame are captured.
[0072] In some aspects, the frame alignment system may receive an estimate of the delay associated with capturing a high exposure setting frame (e.g., a long exposure frame) and a low exposure setting frame (e.g., a short exposure frame) that will be combined to generate a high dynamic range (HDR) output. The shift may be further predicted based on the estimate of the delay. In some aspects, the estimate of the delay may include an average delay between capturing a high exposure setting frame and a short exposure setting frame. In some aspects, the shift is predicted based on a sensor-based alignment library (e.g., the sensor ML system library 804, which may include an RNN library as noted above) generated using a trained machine learning model. The trained machine learning model may include a recurrent neural network.
[0073] At block 1006, the frame alignment system applies a shift to align one or more features of the first frame with one or more features of the second frame. The one or more features of the first frame may be the same as the one or more features of the second frame. For example, a shift may be applied to the first frame having a higher exposure setting to align one or more features of the first frame with one or more features of the second frame.
[0074] At block 1008 , the frame alignment system generates a frame output (eg, an HDR frame output) at least in part by combining at least a portion of the first frame with at least a portion of the second frame, as described herein.
[0075] In some aspects, the frame alignment system accumulates first image sensor data associated with the first frame in a first buffer (e.g., buffer 703) and accumulates second image sensor data associated with the second frame in a second buffer (e.g., buffer 705). The shift based on the first image sensor data can be applied while the first image sensor data is being accumulated in the first buffer.
[0076] In some examples, the processes described herein (e.g., process 1000 and / or other processes described herein) can be performed by a computing device or apparatus. In one example, one or more of the processes can be performed by Figure 3 In another example, one or more of the processes may be performed by Fig.13 The computing system 1300 shown executes. For example, Fig.13 The computing device of the illustrated computing system 1300 may include components of the image capture system 300 and may implement Fig.10 Operations of process 1000 and / or other processes described herein.
[0077] The computing device may include any suitable device, such as a vehicle or a computing device of a vehicle (e.g., a driver monitoring system (DMS) of a vehicle), a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smart watch, or other wearable device), a server computer, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein (including process 1000 and / or any other process described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.
[0078] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.
[0079] Process 1000 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.
[0080] Additionally, process 1000 and / or any other process described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, through hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.
[0081] As noted above, various aspects of the present disclosure may utilize a machine learning model or system. Fig.11 is an illustrative example of a deep learning neural network 1100 that can be used to implement the above-mentioned machine learning-based alignment prediction. Input layer 1120 includes input data. In an illustrative example, input layer 1120 may include data representing pixels of an input video frame. Neural network 1100 includes multiple hidden layers 1122a, 1122b to 1122n. Hidden layers 1122a, 1122b to 1122n include "n" hidden layers, where "n" is an integer greater than or equal to one. Multiple hidden layers can include as many layers as required for a given application. Neural network 1100 also includes an output layer 1121, which provides outputs generated by the processing performed by hidden layers 1122a, 1122b to 1122n. In an illustrative example, output layer 1121 can provide classification of objects in the input video frame. Classification may include categories that identify activity types (e.g., looking up, looking down, closing eyes, yawning, etc.).
[0082] Neural network 1100 is a multi-layer neural network composed of interconnected nodes. Each node can represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains the information as it processes it. In some cases, neural network 1100 may include a feedforward network in which there is no feedback connection in which the output of the network is fed back to itself. In some cases, neural network 1100 may include a recursive neural network, which may have loops that allow information to be carried across nodes when reading in input.
[0083] Information can be exchanged between nodes by node-to-node interconnection between each layer. The nodes of input layer 1120 can activate a group of nodes in the first hidden layer 1122a. For example, as shown in the figure, each input node in the input nodes of input layer 1120 is connected to each node in the nodes of the first hidden layer 1122a. The nodes of the first hidden layer 1122a can transform the information of each input node by applying an activation function to the input node information. The information derived from the transformation can then be passed to the nodes of the next hidden layer 1122b and can activate the nodes of the next hidden layer, and the nodes of the next hidden layer can perform their own specified functions. Example functions include convolution, upsampling, data conversion and / or any other suitable function. Then, the output of hidden layer 1122b can activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 1122n can activate one or more nodes of output layer 1121, and output is provided at the one or more nodes. In some cases, although a node in neural network 1100 (e.g., node 1126) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.
[0084] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 1100. Once neural network 1100 is trained, it may be referred to as a trained neural network, which may be used to classify one or more activities. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have a tunable digital weight that may be tuned (e.g., based on a training data set), thereby allowing neural network 1100 to adapt to the input and be able to learn as more and more data is processed.
[0085] The neural network 1100 is pre-trained to process features from data in the input layer 1120 using different hidden layers 1122a, 1122b to 1122n to provide outputs through the output layer 1121. In examples where the neural network 1100 is used to identify features and / or objects in images, the neural network 1100 may be trained using training data that includes both images and labels, as described above. For example, training images may be input into the network with each training frame having a label indicating a feature in the image (for a feature extraction machine learning system) or a label indicating a class of activity in each frame. In one example using object classification for illustrative purposes, the training frames may include an image of the number 2, in which case the label for the image may be [0 0 1 0 0 0 0 0 0 0].
[0086] In some cases, the neural network 1100 may use a training process known as back propagation to adjust the weights of the nodes. As noted above, the back propagation process may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. This process may be repeated for a certain number of iterations for each set of training images until the neural network 1100 is trained well enough so that the weights of each layer are accurately tuned.
[0087] For the example of identifying features and / or objects in an image, a forward pass may include passing a training image through the neural network 1100. Prior to training the neural network 1100, the weights are initially randomized. As an illustrative example, a frame may include an array of numbers representing pixels of an image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 digital array having 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or a brightness and two chrominance components, etc.).
[0088] As noted above, for the first training iteration of the neural network 1100, the output will likely include values that do not prioritize any particular class due to the weights randomly selected at initialization. For example, if the output is a vector with probabilities that an object includes different classes, the probability values for each of the different classes may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 1100 cannot determine low-level features and therefore cannot accurately determine what the classification of the object may be. A loss function can be used to analyze the errors in the output. Any suitable loss function definition can be used, such as a cross entropy loss. Another example of a loss function includes the mean squared error (MSE), which is defined as (target - output) 2The loss can be set equal to E 总 The value of .
[0089] For the first training image, the loss (or error) will be high because the actual value will be very different from the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 1100 can perform a backward pass by determining which inputs (weights) contribute the most to the loss of the network, and the weights can be adjusted so that the loss decreases and is ultimately minimized. The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes the most to the loss of the network. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as Where w represents the weight, w i Represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes a larger weight update, while a lower value indicates a smaller weight update.
[0090] The neural network 1100 may include any suitable deep network. One example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layers of the CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. The neural network 1100 may include any other deep network other than a CNN, such as an autoencoder, a deep belief network (DBN), a recursive neural network (RNN), etc.
[0091] Fig.12 is an illustrative example of a convolutional neural network (CNN) 1200. The input layer 1220 of the CNN 1200 includes data representing an image or frame. For example, the data may include an array of numbers representing pixels of an image, where each number in the array includes a value from 0 to 255 describing the intensity of the pixel at the location of the number in the array. Using the previous example from above, the array may include a 28×28×3 digital array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luminance and two chrominance components, etc.). The image may be passed through a convolutional hidden layer 1222a, an optional non-linear activation layer, a pooling hidden layer 1222b, and a fully connected hidden layer 1222c to obtain an output at an output layer 1224. Although Fig.12Only one hidden layer among the hidden layers is shown in , but one of ordinary skill will appreciate that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 1200. As previously described, the output may indicate a single category of an object, or may include a probability of a category that best describes an object in the image.
[0092] The first layer of CNN 1200 is a convolutional hidden layer 1222a. The convolutional hidden layer 1222a analyzes the image data of the input layer 1220. Each node of the convolutional hidden layer 1222a is connected to a node (pixel) region in the input image called a receptive field. The convolutional hidden layer 1222a can be considered as one or more filters (each filter corresponds to a different activation or feature map), where each convolution iteration of the filter is a node or neuron of the convolutional hidden layer 1222a. For example, the area of the input image covered by the filter at each convolution iteration will be the receptive field of the filter. In an illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in the convolutional hidden layer 1222a. Each connection between a node and the receptive field of that node learns a weight, and in some cases learns an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node of hidden layer 1222a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has an array of weights (numbers) and the same depth as the input. For the video frame example, the filter will have a depth of 3 (according to the three color components of the input image). An illustrative example size of the filter array is 5×5×3, corresponding to the size of the node's receptive field.
[0093] The convolutional nature of the convolutional hidden layer 1222a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1222a may start at the upper left corner of the input image array and may be convolved around the input image. As noted above, each convolution iteration of the filter may be considered as a node or neuron of the convolutional hidden layer 1222a. In each convolution iteration, the value of the filter is multiplied by the original pixel values of the corresponding number of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values at the upper left corner of the input image array). The multiplications from each convolution iteration may be added together to obtain the sum of the iteration or node. Next, the process is continued at the next position in the input image according to the receptive field of the next node in the convolutional hidden layer 1222a. For example, the filter may be moved by a step amount (referred to as a stride) to the next receptive field. The stride may be set to 1 or other suitable amount. For example, if the stride is set to 1, the filter will move 1 pixel to the right at each convolution iteration. Processing the filter at each unique location in the input volume produces a number representing the filter result at that location, thereby determining a summed value for each node of the convolutional hidden layer 1222a.
[0094] The mapping from the input layer to the convolutional hidden layer 1222a is called an activation map (or feature map). The activation map includes the value of each node, which represents the filter result at each location of the input volume. The activation map may include an array that includes various summed values produced by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will include a 24×24 array. The convolutional hidden layer 1222a may include several activation maps in order to identify multiple features in the image. Fig.12 The example shown includes three activation maps. Using the three activation maps, the convolutional hidden layer 1222a can detect three different types of features, each of which is detectable across the entire image.
[0095] In some examples, a nonlinear hidden layer may be applied after the convolutional hidden layer 1222a. A nonlinear layer may be used to introduce nonlinearity into a system that already computes linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer may apply the function f(x)=max(0,x) to all values in the input volume, which changes all negative activations to 0. Thus, the ReLU may increase the nonlinear properties of the CNN 1200 without affecting the receptive field of the convolutional hidden layer 1222a.
[0096] The pooling hidden layer 1222b may be applied after the convolutional hidden layer 1222a (and after the nonlinear hidden layer when used). The pooling hidden layer 1222b is used to simplify the information in the output from the convolutional hidden layer 1222a. For example, the pooling hidden layer 1222b may take each activation map output from the convolutional hidden layer 1222a and generate a condensed activation map (or feature map) using a pooling function. Max pooling is an example of a function performed by a pooling hidden layer. The pooling hidden layer 1222a uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. A pooling function (e.g., a maximum pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1222a. In Fig.12 In the example shown in , three pooling filters are used for the three activation maps in the convolutional hidden layer 1222a.
[0097] In some examples, max pooling may be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as a stride of 2) to the activation map output from the convolutional hidden layer 1222a. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer may summarize an area of 2×2 nodes in the previous layer (where each node is a value in the activation map). For example, four values (nodes) in the activation map will be analyzed by a 2×2 max pooling filter at each iteration of the filter, where the maximum of the four values is output as the "maximum" value. If such a max pooling filter is applied to an activation filter having a dimension of 24×24 nodes from the convolutional hidden layer 1222a, the output from the pooling hidden layer 1222b will be an array of 12×12 nodes.
[0098] In some examples, an L2-norm pooling filter may also be used. The L2-norm pooling filter involves calculating the square root of the sum of the squares of the values in a 2×2 region (or other suitable region) of the activation map (rather than calculating the maximum value as done in max pooling), and using the calculated value as output.
[0099] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found anywhere in a region of an image, and discards the exact location information. This can be done without affecting the results of feature detection, because once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (and other pooling methods) provides the following benefits: far fewer features are pooled, thereby reducing the number of parameters required in subsequent layers of CNN 1200.
[0100] The last layer of connections in the network is a fully connected layer that connects each node from the pooling hidden layer 1222b to each of the output nodes in the output layer 1224. Using the above example, the input layer includes 28×28 nodes that encode the pixel intensities of the input image, the convolutional hidden layer 1222a includes 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for the filter) to the three activation maps, and the pooling hidden layer 1222b includes a layer of 3×12×12 hidden feature nodes based on applying a max pooling filter to a 2×2 region on each of the three feature maps. Extending this example, the output layer 1224 may include ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 1222b is connected to each node of the output layer 1224.
[0101] The fully connected layer 1222c can obtain the output of the previous pooled hidden layer 1222b (which should represent an activation map of high-level features) and determine the features most relevant to a particular category. For example, the fully connected layer 1222c layer can determine the high-level features that are most strongly associated with a particular category and can include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 1222c and the pooled hidden layer 1222b can be calculated to obtain probabilities for different categories. For example, if the CNN 1200 is used to predict that the object in the video frame is a person, there will be high values in the activation map representing the high-level features of the person (e.g., there are two legs, there is a face at the top of the object, there are two eyes at the upper left and upper right of the face, there is a nose in the middle of the face, there is a mouth at the bottom of the face, and / or other features common to people).
[0102] In some examples, the output from the output layer 1224 may include an M-dimensional vector (in the previous example, M=10). M indicates the number of categories that the CNN 1200 must choose from when classifying objects in the image. Other example outputs may also be provided. Each number in the M-dimensional vector may represent the probability that the object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector represents that objects of ten different categories are [0 0 0.05 0.8 0 0.15 0 0 0 0], the vector indicates that the image is a 5% probability of an object of the third category (e.g., a dog), the image is an 80% probability of an object of the fourth category (e.g., a person), and the image is a 15% probability of an object of the sixth category (e.g., a kangaroo). The probability for a category can be considered as the confidence level that the object is part of that category.
[0103] Fig.13 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.13An example of a computing system 1300 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1305. The connection 1305 may be a physical connection using a bus, or a direct connection into the processor 1310, such as in a chipset architecture. The connection 1305 may also be a virtual connection, a networked connection, or a logical connection.
[0104] In some embodiments, computing system 1300 is a distributed system, where the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some embodiments, a component may be a physical device or a virtual device.
[0105] The example system 1300 includes at least one processing unit (CPU or processor) 1310 and connections 1305 that couple various system components including system memory 1315, such as read only memory (ROM) 1320 and random access memory (RAM) 1325, to the processor 1310. The computing system 1300 may include a cache 1312 of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 1310.
[0106] Processor 1310 may include any general purpose processor and hardware or software services, such as services 1332, 1334, and 1336 stored in storage device 1330, that are configured to control processor 1310 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1310 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.
[0107] To enable user interaction, the computing system 1300 includes an input device 1345 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1300 may also include an output device 1335 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1300. The computing system 1300 may include a communication interface 1340, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transfer, Low Energy (BLE) wireless signal transfer, Wireless signal transfer, radio frequency identification (RFID) wireless signal transfer, near field communication (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 802.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transfer, public switched telephone network (PSTN) signal transfer, integrated services digital network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communication interface 1340 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1300 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.
[0108] The storage device 1330 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.
[0109] Storage device 1330 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1310, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1310, connector 1305, output device 1335, etc.
[0110] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.
[0111] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.
[0112] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments may be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form in order to avoid confusing these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown in order to avoid confusing each embodiment without necessary details.
[0113] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.
[0114] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. These instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the examples described include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.
[0115] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.
[0116] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.
[0117] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the scope of this specification, the embodiments can be used in any number of environments and applications beyond the environments and applications described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For illustrative purposes, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.
[0118] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.
[0119] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.
[0120] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).
[0121] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.
[0122] Various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician can implement the functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.
[0123] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including program codes, which include instructions that execute one or more of the above methods when executed. A computer-readable data storage medium may form a part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.
[0124] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein.
[0125] Illustrative aspects of the present disclosure include:
[0126] Aspect 1. A device for frame alignment, comprising: a memory; and at least one processor, the at least one processor coupled to the memory and configured to: receive sensor data indicating movement associated with an image capture device that captures a first frame and a second frame, wherein the first frame is associated with a higher exposure setting than the second frame; predict a shift based on the sensor data to align one or more features of the first frame with one or more features of the second frame; apply the shift to align the one or more features of the first frame with the one or more features of the second frame; and generate a frame output at least in part by combining at least a portion of the first frame with at least a portion of the second frame.
[0127] Aspect 2. The apparatus according to aspect 1, wherein the shift is predicted before the first frame and the second frame are captured.
[0128] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein the at least one processor is configured to: receive an estimate of a delay associated with capturing a high exposure settings frame and a low exposure settings frame, wherein the high exposure settings frame and the low exposure settings frame are to be combined to generate a high dynamic range (HDR) output; and predict the shift based on the estimate of the delay.
[0129] Aspect 4. An apparatus according to any one of aspects 1 to 3, wherein the first frame is associated with a higher exposure setting than the second frame comprises: the first frame is associated with a longer exposure time than the second frame.
[0130] Aspect 5. An apparatus according to any one of Aspects 1 to 4, wherein the at least one processor is configured to: predict the shift based on a sensor-based alignment library generated using a trained machine learning model.
[0131] Aspect 6. An apparatus according to Aspect 5, wherein the trained machine learning model comprises a recurrent neural network.
[0132] Aspect 7. An apparatus according to any one of Aspects 1 to 6, wherein, in order to apply the shift, the at least one processor is configured to: apply the shift to the first frame having the higher exposure setting so that the one or more features of the first frame are aligned with the one or more features of the second frame.
[0133] Clause 8. An apparatus according to any one of Clauses 1 to 7, wherein the frame output comprises a high dynamic range (HDR) frame output.
[0134] Aspect 9. An apparatus according to any one of Aspects 1 to 8, wherein the at least one processor is configured to: accumulate first image sensor data associated with the first frame in a first buffer; accumulate second image sensor data associated with the second frame in a second buffer; and apply the shift based on the first image sensor data while the first image sensor data is being accumulated in the first buffer.
[0135] Clause 10. The apparatus of clause 9, wherein at least one of the first image sensor data or the second image sensor data is accumulated from a center position of the first buffer or the second buffer, respectively.
[0136] Aspect 11. An apparatus according to any one of aspects 1 to 10, wherein the sensor data includes gyroscope data.
[0137] Aspect 12. A method for frame alignment, comprising: receiving sensor data indicating movement associated with an image capture device that captures a first frame and a second frame, wherein the first frame is associated with a higher exposure setting than the second frame; predicting a shift based on the sensor data to align one or more features of the first frame with one or more features of the second frame; applying the shift to align the one or more features of the first frame with the one or more features of the second frame; and generating a frame output at least in part by combining at least a portion of the first frame with at least a portion of the second frame.
[0138] Clause 13. The method of clause 12, wherein the shift is predicted before the first frame and the second frame are captured.
[0139] Aspect 14. The method according to any one of Aspects 12 to 13 further includes: receiving an estimate of a delay associated with capturing a high exposure setting frame and a low exposure setting frame, wherein the high exposure setting frame and the low exposure setting frame are to be combined to generate a high dynamic range (HDR) output; and predicting the shift based on the estimate of the delay.
[0140] Clause 15. The method according to any one of clauses 12 to 14, wherein the first frame being associated with a higher exposure setting than the second frame comprises: the first frame being associated with a longer exposure time than the second frame.
[0141] Aspect 16. The method according to any one of Aspects 12 to 15, further comprising: predicting the shift based on a sensor-based alignment library generated using a trained machine learning model.
[0142] Aspect 17. A method according to Aspect 16, wherein the trained machine learning model comprises a recursive neural network.
[0143] Aspect 18. A method according to any one of Aspects 12 to 17, wherein applying the shift comprises: applying the shift to the first frame having the higher exposure setting so that the one or more features of the first frame are aligned with the one or more features of the second frame.
[0144] Aspect 19. The method of any one of aspects 12 to 18, wherein the frame output comprises a high dynamic range (HDR) frame output.
[0145] Aspect 20. The method according to any one of Aspects 12 to 19 further includes: accumulating first image sensor data associated with the first frame in a first buffer; accumulating second image sensor data associated with the second frame in a second buffer; and applying the shift based on the first image sensor data while the first image sensor data is being accumulated in the first buffer.
[0146] Aspect 21. The method of aspect 20, wherein at least one of the first image sensor data or the second image sensor data is accumulated from a center position of the first buffer or the second buffer, respectively.
[0147] Aspect 22. A method according to any one of aspects 12 to 21, wherein the sensor data includes gyroscope data.
[0148] Aspect 23. A computer-readable medium comprising: at least one instruction for causing a computer or a processor to perform the operations according to any one of Aspects 1 to 22.
[0149] Aspect 24. An apparatus for model generation, the apparatus comprising means for performing the operations according to any one of aspects 1 to 22.
[0150] Aspect 25. An apparatus for model generation, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to perform the operations according to any one of aspects 1 to 22.
Claims
1. A device for frame alignment, comprising: Memory; and at least one processor coupled to the memory and configured to: receiving sensor data indicative of movement associated with an image capture device capturing a first frame and a second frame, wherein the first frame is associated with a higher exposure setting than the second frame; predicting, based on the sensor data, a shift for aligning one or more features of the first frame with one or more features of the second frame; applying the shift to align the one or more features of the first frame with the one or more features of the second frame; and A frame output is generated at least in part by combining at least a portion of the first frame with at least a portion of the second frame. 2 . The apparatus of claim 1 , wherein the shift is predicted before the first frame and the second frame are captured.
3. The apparatus of claim 1 , wherein the at least one processor is configured to: receiving an estimate of a delay associated with capturing a high exposure settings frame and a low exposure settings frame to be combined for generating a high dynamic range (HDR) output; and The shift is predicted based on the estimate of the delay.
4. The apparatus of claim 1 , wherein the first frame being associated with a higher exposure setting than the second frame comprises: The first frame is associated with a longer exposure time than the second frame.
5. The apparatus of claim 1, wherein the at least one processor is configured to predict the shift based on a sensor-based alignment library generated using a trained machine learning model.
6. The device of claim 5, wherein the trained machine learning model comprises a recurrent neural network.
7. The apparatus of claim 1, wherein to apply the shift, the at least one processor is configured to: apply the shift to the first frame having the higher exposure setting to align the one or more features of the first frame with the one or more features of the second frame.
8. The apparatus of claim 1, wherein the frame output comprises a high dynamic range (HDR) frame output.
9. The apparatus of claim 1 , wherein the at least one processor is configured to: accumulating first image sensor data associated with the first frame in a first buffer; accumulating second image sensor data associated with the second frame in a second buffer; and The shift based on the first image sensor data is applied while the first image sensor data is being accumulated in the first buffer.
10. The device of claim 9, wherein at least one of the first image sensor data or the second image sensor data is accumulated from a center position of the first buffer or the second buffer, respectively.
11. The apparatus of claim 1, wherein the sensor data comprises gyroscope data.
12. A method for frame alignment, comprising: receiving sensor data indicative of movement associated with an image capture device capturing a first frame and a second frame, wherein the first frame is associated with a higher exposure setting than the second frame; predicting, based on the sensor data, a shift for aligning one or more features of the first frame with one or more features of the second frame; applying the shift to align the one or more features of the first frame with the one or more features of the second frame; as well as A frame output is generated at least in part by combining at least a portion of the first frame with at least a portion of the second frame.
13. The method of claim 12, wherein the shift is predicted before the first frame and the second frame are captured.
14. The method according to claim 12, further comprising: receiving an estimate of a delay associated with capturing a high exposure settings frame and a low exposure settings frame to be combined for generating a high dynamic range (HDR) output; and The shift is predicted based on the estimate of the delay.
15. The method of claim 12, wherein associating the first frame with a higher exposure setting than the second frame comprises: The first frame is associated with a longer exposure time than the second frame.
16. The method according to claim 12, further comprising: The shifts are predicted based on a sensor-based alignment library generated using a trained machine learning model.
17. The method of claim 16, wherein the trained machine learning model comprises a recurrent neural network.
18. The method of claim 12, wherein applying the shift comprises: The shift is applied to the first frame having the higher exposure setting to align the one or more features of the first frame with the one or more features of the second frame.
19. The method of claim 12, wherein the frame output comprises a high dynamic range (HDR) frame output.
20. The method of claim 12, further comprising: accumulating first image sensor data associated with the first frame in a first buffer; accumulating second image sensor data associated with the second frame in a second buffer; as well as The shift based on the first image sensor data is applied while the first image sensor data is being accumulated in the first buffer.
21. The method of claim 20, wherein at least one of the first image sensor data or the second image sensor data is accumulated from a center position of the first buffer or the second buffer, respectively.
22. The method of claim 12, wherein the sensor data comprises gyroscope data.
23. A non-transitory computer readable medium having instructions which, when executed by one or more processors, cause the one or more processors to: receiving sensor data indicative of movement associated with an image capture device capturing a first frame and a second frame, wherein the first frame is associated with a higher exposure setting than the second frame; predicting, based on the sensor data, a shift for aligning one or more features of the first frame with one or more features of the second frame; applying the shift to align the one or more features of the first frame with the one or more features of the second frame; as well as A frame output is generated at least in part by combining at least a portion of the first frame with at least a portion of the second frame.
Citation Information
Patent Citations
Techniques for convolutional neural network-based multi-exposure fusion of multiple image frames and for deblurring multiple image frames
CN113454981A
Image processing method
US20080253758A1
