Image processing based on object classification

Abstract

Examples are described of applying different settings for image capture to different portions of image data. For example, an image sensor can capture image data of a scene and can send the image data to an image signal processor (ISP) and a classification engine for processing. The classification engine can determine that a first object image region depicts a first object class and a second object image region depicts a second object class. Different confidence regions of the image data can identify different degrees of confidence in the classification. The ISP can generate an image by applying different settings to different portions of the image data. The different portions of the image data can be identified based on the object image regions and the confidence regions.

Description

[0001] This application is a divisional application of patent application No. 202180055549.9, filed on July 20, 2021, entitled "Image Processing Based on Object Classification". Technical Field

[0002] In summary, this application relates to image capture and image processing. More specifically, this application relates to a system and method for automatically guiding image processing of photographs based on the classification of objects in a shooting scene. Background Technology

[0003] Image capture devices use image sensors with an array of photodiodes to capture images from initial light from a scene. An image signal processor (ISP) then processes the raw image data captured by the photodiodes of the image sensor into an image that can be stored and viewed by the user. How the scene is depicted in the image depends in part on capture settings that control how much light the image sensor receives, such as exposure time and aperture settings. How the scene is depicted in the image also depends on how the ISP is tuned to process the photodiode data captured by the image sensor into an image.

[0004] Traditionally, the image capture device's ISP (Image Signal Processor) is tuned only once during manufacturing. ISP tuning affects how each image is processed within the image capture device, and impacts every pixel of each image. Users typically expect image capture devices to capture high-quality images regardless of the scene being shot. To avoid situations where the image capture device cannot properly capture certain types of scenes, the ISP is usually tuned to be reasonably applicable to as many scene types as possible. However, precisely because of this, tuning a traditional ISP is generally not optimal for shooting all types of scenes. Summary of the Invention

[0005] This paper describes systems and techniques for determining and applying different ISP settings for different image regions. In some examples, image capture and processing devices can use different ISP settings for different image regions to process raw image data captured by an image sensor. In some cases, a classification engine can divide raw image data into different object image regions based on the detection of different object types within different image regions in the raw image data. By applying different ISP settings to different regions in an image, the ISP is optimized for depicting the object type in the image. In an illustrative example, the ISP can use ISP settings that enhance the sharpness in regions depicting human hair in an image, which can enhance the texture clarity of the hair. Within the same image, the ISP can use different ISP settings that reduce the sharpness in regions depicting human skin in an image and enhance noise reduction, which can result in a processed image depicting smoother skin. Different confidence regions of image data can identify different levels of confidence in the classification. Settings can be further modified based on confidence. The intensity of a particular ISP parameter (such as noise reduction, sharpness, color saturation, or tone mapping) can be adjusted from default values ​​for that pixel based on the object category depicted at the pixel and the confidence level of that classification. For example, the increase or decrease of a default value associated with a particular object category can be moderated (if the confidence level of that category is low), while it can be increased if the confidence level of that category is high.

[0006] In one example, an apparatus for data encoding is provided. The apparatus includes: a memory; and one or more processors (e.g., implemented in a circuit system) coupled to the memory. The one or more processors are configured to: receive image data captured by an image sensor; determine that a first object image region in the image data depicts a first object category among a plurality of object categories; determine that a second object image region in the image data depicts a second object category among the plurality of object categories; identify a plurality of confidence levels corresponding to a plurality of confidence image regions in the image data, wherein each of the plurality of confidence levels identifies a confidence level regarding the depiction of one object category among the plurality of object categories by the corresponding confidence image region in the plurality of confidence image regions; and generate an image based on the image data using an image capture process, including by applying different settings for the image capture process to different portions of the image data, the different portions of the image data being identified based on the first object image region, the second object image region, and the plurality of confidence image regions.

[0007] In another example, an image processing method is provided. The method includes: receiving image data captured by an image sensor. The method includes: determining that a first object image region in the image data depicts a first object category among a plurality of object categories. The method includes: determining that a second object image region in the image data depicts a second object category among the plurality of object categories. The method includes: identifying a plurality of confidence levels corresponding to a plurality of confidence image regions in the image data, wherein each of the plurality of confidence levels identifies a confidence level regarding the depiction of one of the plurality of object categories by the corresponding confidence image region among the plurality of confidence image regions. The method includes: generating an image based on the image data using an image capture process, including by applying different settings for the image capture process to different portions of the image data, the different portions of the image data being identified based on the first object image region, the second object image region, and the plurality of confidence image regions.

[0008] In another example, a non-transitory computer-readable storage medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: receive image data captured by an image sensor; determine that a first object image region in the image data depicts a first object category among a plurality of object categories; determine that a second object image region in the image data depicts a second object category among the plurality of object categories; identify a plurality of confidence levels corresponding to a plurality of confidence image regions in the image data, wherein each of the plurality of confidence levels identifies a confidence level regarding the depiction of one of the plurality of object categories by the corresponding confidence image region among the plurality of confidence image regions; and generate an image based on the image data using an image capture process, including by applying different settings for the image capture process to different portions of the image data, the different portions of the image data being identified based on the first object image region, the second object image region, and the plurality of confidence image regions.

[0009] In another example, an apparatus for image processing is provided. The apparatus includes: a unit for receiving image data captured by an image sensor. The apparatus includes: a unit for determining that a first object image region in the image data depicts a first object category among a plurality of object categories. The apparatus includes: a unit for determining that a second object image region in the image data depicts a second object category among the plurality of object categories. The apparatus includes: a unit for identifying a plurality of confidence levels corresponding to a plurality of confidence image regions in the image data, wherein each of the plurality of confidence levels identifies a confidence level regarding whether the corresponding confidence image region among the plurality of confidence image regions depicts one of the plurality of object categories. The apparatus includes: a unit for generating an image based on the image data using an image capture process, including by applying different settings for the image capture process to different portions of the image data, the different portions of the image data being identified based on the first object image region, the second object image region, and the plurality of confidence image regions.

[0010] In some aspects, the methods, apparatus, and computer-readable media described above further include: generating one or more modification amounts, said one or more modification amounts identifying at least one of the following: a first deviation for the first object image region from a default setting for the image capture process, and a second deviation for the second object image region from the default setting for the image capture process, wherein the different settings for the image capture process are based on said one or more modification amounts. In some aspects, the methods, apparatus, and computer-readable media described above further include: adjusting said one or more modification amounts, including mixing said one or more modification amounts with a mixed update based on a plurality of confidence levels corresponding to said plurality of confidence image regions, wherein mixing said one or more modification amounts with the mixed update is used to adjust at least one of the first deviation and the second deviation in at least one region of the image data.

[0011] In some aspects, the above-described methods, apparatus, and computer-readable media further include: generating a category map that divides the image data into a plurality of object image regions including a first object image region and a second object image region, wherein each of the plurality of object image regions corresponds to one of the plurality of object categories; identifying the first object category corresponding to a first setting for the image capture process; and identifying the second object category corresponding to a second setting for the image capture process. In some aspects, the above-described methods, apparatus, and computer-readable media further include: generating a confidence map that divides the image data into a plurality of confidence image regions corresponding to the plurality of confidence levels, wherein the different portions of the image data are identified based on the category map and the confidence map.

[0012] In some aspects, the image capture process includes processing the image data using an image signal processor (ISP) of the one or more processors, wherein the different settings for the image capture process are different tuning settings for the ISP. In some aspects, the different tuning settings for the ISP include applying different intensities of ISP tuning parameters during processing of the image data using the ISP, wherein the ISP tuning parameters are one of noise reduction, sharpening, color saturation, color mapping, color processing, and tone mapping. In some aspects, the different settings include settings associated with at least one of: lens position, flash, focus, exposure, white balance, aperture size, shutter speed, ISO, analog gain, digital gain, noise reduction, sharpening, tone mapping, color saturation, demosaic, color space conversion, colorization, edge enhancement, image combination for high dynamic range (HDR), effects, artificial noise addition, edge-oriented magnification, magnification, reduction, and electronic image stabilization. In some aspects, the above-described methods, apparatus, and computer-readable media further include: processing the image data, including at least one of the following operations: depixelating the image data and converting the image data from a first color space to a second color space.

[0013] In some aspects, the methods, apparatus, and computer-readable media described above further include: receiving user input associated with at least one of the first object image region and the second object image region, wherein at least one of the different settings is defined based on the user input and corresponds to one of the first object image region and the second object image region. In some aspects, applying the different settings for the image capture process to the different portions of the image data includes: using an image signal processor (ISP) to apply the different settings for the image capture process to the different portions of the image data. In some aspects, identifying the first object image region and the second object image region includes: using a classification engine to identify the first object image region and the second object image region, the classification engine being at least partially located on an integrated circuit chip. In some aspects, the methods, apparatus, and computer-readable media described above further include: displaying the image on a display.

[0014] In some aspects, the apparatus includes a camera, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wireless communication device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computing device, or other devices. In some aspects, the one or more processors include an image signal processor (ISP). In some aspects, the apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus includes an image sensor for capturing the image data. In some aspects, the apparatus also includes a display for displaying the images, one or more notifications (e.g., notifications associated with processing the images), and / or other displayable data.

[0015] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to define the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of the entire specification, any or all of the drawings, and each claim.

[0016] The foregoing, as well as other features and embodiments, will become more apparent upon reference to the following description, claims, and drawings. Attached Figure Description

[0017] The illustrative embodiments of this application are described in detail below with reference to the accompanying drawings:

[0018] Figure 1 This is a block diagram illustrating the architecture of an image capture and processing device;

[0019] Figure 2 This is a concept diagram illustrating image processing using category maps and confidence maps;

[0020] Figure 3 This is a conceptual diagram illustrating an image signal processor (ISP) pipeline for image processing based on object classification;

[0021] Figure 4 This is a conceptual diagram illustrating the graph decoder pipeline;

[0022] Figure 5A This is a conceptual diagram illustrating the application of the modification amount at the ISP module, where the modification amount is applied as a multiplier;

[0023] Figure 5B This is a conceptual diagram illustrating the application of modification amounts at the ISP module, where the modification amounts are applied as offsets;

[0024] Figure 5C This is a conceptual diagram illustrating the application of modifications at the ISP module, where the modifications are applied using parameter-based logic;

[0025] Figure 6 This is a conceptual diagram illustrating visual image artifacts introduced by anomalies during image segmentation into image regions during the generation of a category map;

[0026] Figure 7 This is a conceptual diagram illustrating the smooth transition graph processor pipeline;

[0027] Figure 8 This is a conceptual diagram illustrating the use of a smooth transition graph processor to smooth the amount of modification corresponding to an image region;

[0028] Figure 9 This is a diagram showing the Classification Diagram Amplifier (CMUS) pipeline;

[0029] Figure 10 This is a graph showing a comparison between a class map magnified using nearest neighbor magnification and the same class map magnified using nearest neighbor magnification modified with spatial weighted filtering (which is applied using class map magnification (CMUS)).

[0030] Figure 11 This is a conceptual diagram showing an example resolution of image data corresponding to the category map during zoom-out and zoom-in operations;

[0031] Figure 12A This is a flowchart illustrating image processing techniques;

[0032] Figure 12B This is a flowchart illustrating image processing techniques;

[0033] Figure 13This is a flowchart illustrating the transition smoothing technique;

[0034] Figure 14 This is a flowchart illustrating image magnification techniques; and

[0035] Figure 15 This is a diagram illustrating an example of a system used to implement certain aspects of the techniques described in this paper. Detailed Implementation

[0036] Certain aspects and embodiments of this disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments can be applied independently, and some can be applied in combination. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of this application. However, it will be apparent that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting.

[0037] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of these exemplary embodiments will provide those skilled in the art with a feasible description for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0038] An image capture device (e.g., a camera) is a device that uses an image sensor to receive light and capture image frames (such as still images or video frames). The terms "image," "image frame," and "frame" are used interchangeably herein. An image capture device typically includes at least one lens that receives light from a scene and directs the light toward the image sensor of the image capture device. The light received by the lens passes through an aperture controlled by one or more control mechanisms and is received by the image sensor. The one or more control mechanisms may control exposure, focus, and / or zoom based on information from the image sensor and / or based on information from an image processor (e.g., a host or application process and / or an image signal processor). In some examples, the one or more control mechanisms include a motor or other control mechanism that moves the lens of the image capture device to a target lens position.

[0039] As described in more detail below, this document describes systems and techniques for determining different settings for the image capture process and applying them to different image regions from image data from an image sensor. In some examples, the image capture and processing apparatus may use different settings for different image regions to process image data captured by the image sensor. The image data may be raw image data or data that has been partially processed by an image signal processor (ISP) or other components. For example, raw image data or partially processed image data may be processed by the ISP using demosaicing, color space conversion, and / or another processing operation discussed herein.

[0040] In some cases, a classification engine can segment image data into different image regions based on the detection of different types of objects within different image regions. Images are generated by applying different settings to different regions of the image data, where the capture and / or processing of the image data in the image is optimized for each type of object depicted in the image. In some examples, these settings may be associated with certain ISP tuning parameters of the ISP. In an illustrative example, the ISP can process the image data using ISP settings that enhance the sharpness in the region depicting human hair, thus enhancing the texture clarity of the hair in the processed image. The ISP can use different ISP settings that reduce sharpness in different regions depicting human skin in the image data and enhance noise reduction. Different ISP settings can result in a processed image depicting smooth skin while also depicting sharp and textured hair. In some examples, these settings can be applied to image capture settings such as focus, exposure time, aperture size, ISO, flash, any combination thereof, and / or other image capture settings discussed herein. In some examples, these settings can be applied as post-processing settings after the ISP has converted the image data from the raw image data from the image sensor into an image. Post-processing settings can include adjustments to brightness, contrast, saturation, hue level, histogram, any combination thereof, and / or other processing settings discussed herein.

[0041] Figure 1This is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., an image of scene 110). The image capture and processing system 100 can capture individual images (or photographs) and / or can capture video comprising multiple images (or video frames) in a specific sequence. A lens 115 of the system 100 faces scene 110 and receives light from scene 110. The lens 115 refracts the light toward an image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0042] One or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or information from image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components; for example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms besides those shown, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture characteristics.

[0043] The focus control mechanism 125B of the control mechanism 120 can obtain focus settings. In some examples, the focus control mechanism 125B stores the focus settings in a memory register. Based on the focus settings, the focus control mechanism 125B can adjust the position of the lens 115 relative to the image sensor 130. For example, based on the focus settings, the focus control mechanism 125B can adjust the focus by driving a motor or servo device (or other lens mechanism) to move the lens 115 closer to or further away from the image sensor 130. In some cases, additional lenses may be included in the system 100, such as one or more microlenses on each photodiode of the image sensor 130, each of which refracts light received from the lens 115 toward the corresponding photodiode before the light reaches it. The focus settings can be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus settings can be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings.

[0044] The exposure control mechanism 125A of the control mechanism 120 can obtain the exposure settings. In some cases, the exposure control mechanism 125A stores the exposure settings in a memory register. Based on the exposure settings, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or aperture order), the duration of the aperture opening (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure settings may be referred to as image capture settings and / or image processing settings.

[0045] The zoom control mechanism 125C of the control mechanism 120 can obtain zoom settings. In some examples, the zoom control mechanism 125C stores the zoom settings in a memory register. Based on the zoom settings, the zoom control mechanism 125C can control the focal length of an assembly of lens elements (lens assembly) including lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by driving one or more motors or servo devices (or other lens mechanisms) to move one or more of these lenses relative to each other. The zoom settings may be referred to as image capture settings and / or image processing settings. In some examples, the lens assembly may include a parfocal zoom lens or a zoom-length zoom lens. In some examples, the lens assembly may include a focusing lens (which in some cases may be lens 115) that first receives light from scene 110, wherein the light then passes through a focusless zoom system between the focusing lens (e.g., lens 115) and image sensor 130 before reaching image sensor 130. In some cases, a focusless zoom system may include two positive (e.g., converging, convex) lenses having equal or similar focal lengths (e.g., within a threshold difference from each other), with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control mechanism 125C moves one or more lenses in the focusless zoom system, such as one or both of the positive lenses and the negative lens.

[0046] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus light matching the color of the color filter covering the photodiode can be measured. For example, Bayer color filters include red, blue, and green color filters, where each pixel of the image is generated based on red light data from at least one photodiode covered in the red color filter, blue light data from at least one photodiode covered in the blue color filter, and green light data from at least one photodiode covered in the green color filter. Instead of or in addition to red, blue, and / or green color filters, other types of color filters may be used, such as yellow, magenta, and / or cyan (also known as "emerald green") color filters. Some image sensors (e.g., image sensor 130) may be completely devoid of color filters and may instead use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes in the pixel array can have different spectral sensitivity profiles, thus responding to light of different wavelengths. Monochrome image sensors may also lack color filters, and therefore lack color depth.

[0047] In some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that prevents light from reaching certain photodiodes or portions of certain photodiodes that can be used for phase detection autofocus (PDAF) at certain times and / or from certain angles. Image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output from the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output from the photodiodes (and / or the signal amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more of the control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide-semiconductor (CMOS), an N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0048] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or any other type of processor 1110 discussed with respect to computing device 1100. Host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) including host processor 152 and ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, and connectivity components (e.g., Bluetooth). TM The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Integrated Circuit 2 (I2C) interface, an Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a Serial General Purpose Input / Output (GPIO) interface, a Mobile Industrial Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High Performance Bus (AHB) bus, any combination thereof), and / or other input / output ports. In an illustrative example, the host processor 152 may use the I2C port to communicate with the image sensor 130, and the ISP 154 may use the MIPI port to communicate with the image sensor 130.

[0049] Image processor 150 can perform multiple tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or a combination thereof. Image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 / 1020, read-only memory (ROM) 145 / 1025, cache, memory units, another storage device, or a combination thereof.

[0050] Various input / output (I / O) devices 160 can be connected to the image processor 150. I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, touchpad, touch-sensitive surface, printer, any other output device 1035, any other input device 1045, or some combination thereof. In some cases, descriptive text can be entered into the image processing device 105B via the physical keyboard or keypad of the I / O device 160, or via the virtual keyboard or keypad of the touchscreen of the I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors that enable wired connections between the system 100 and one or more peripheral devices, through which the system 100 can receive data from and / or send data to one or more peripheral devices. I / O 160 may include one or more wireless transceivers that enable wireless connections between the system 100 and one or more peripheral devices, through which the system 100 can receive data from and / or send data to one or more peripheral devices. Peripheral devices may include any of the types of I / O devices 160 discussed earlier, and they can be considered I / O devices 160 in themselves once they are coupled to ports, jacks, wireless transceivers or other wired and / or wireless connectors.

[0051] In some cases, the image capture and processing system 100 may be a single device. In other cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0052] like Figure 1 As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, namely image capture device 105A and image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, certain components shown in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0053] Image capture and processing system 100 may include electronic devices such as mobile or landline handsets (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, image capture and processing system 100 may include one or more wireless transceivers for wireless communication (such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof). In some implementations, image capture device 105A and image processing device 105B may be different devices. For example, image capture device 105A may include a camera device, and image processing device 105B may include a computing device, such as a mobile handset, desktop computer, or other computing device.

[0054] Although the image capture and processing system 100 is shown to include certain components, those skilled in the art will understand that the image capture and processing system 100 may include components compatible with those in... Figure 1 The components shown are more numerous than those listed herein. Components of the image capture and processing system 100 may include software, hardware, or a combination of one or more software and hardware. For example, in some implementations, components of the image capture and processing system 100 may include electronic circuitry or other electronic hardware (which may include one or more programmable circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable circuits)) and / or may be implemented using electronic circuitry or other electronic hardware, and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 100.

[0055] The ISP is tuned by selecting settings for multiple ISP tuning parameters. These settings can be referred to as ISP settings, ISP tuning settings, ISP tuning parameter settings, tuning settings, tuning parameter settings, or some combination thereof. The ISP uses the settings selected for the ISP tuning parameters to process images. Tuning the ISP is a computationally expensive process, and therefore the ISP is typically tuned only once during manufacturing using fixed adjustment techniques. The settings for the ISP tuning parameters are generally not changed after manufacturing and are therefore applied globally to every pixel of every image processed by the ISP. To avoid situations where image capture devices cannot properly capture certain types of scenes, ISP tuning is usually chosen to be reasonably applicable to as many scene types as possible. However, precisely because of this, adjustments to a conventional ISP are usually not optimal for shooting any type of scene. Therefore, conventional ISP tuning makes a conventional ISP jack-of-all-trades, master of none.

[0056] While ISP tuning is computationally expensive, it is possible to generate multiple settings for certain ISP tuning parameters during manufacturing. For example, for ISP tuning parameters such as sharpness, a high sharpness setting might correspond to an increased level of sharpness, while a low sharpness setting might correspond to a decreased level of sharpness. Different settings can be useful when the image being captured primarily depicts a single type of object (such as a close-up image of a plant, a human face, a vehicle, or food). For images of human faces, a low sharpness setting can be selected via the user interface, or automatically based on the detection of the face in the preview image, to depict smoother facial skin. For images of plants, a high sharpness setting can be selected via the user interface, or automatically based on the detection of the plant in the preview image, to depict more detail in the texture of the leaves and flowers. However, images depicting only one type of object are rare, as most images depict many types of objects. For images depicting multiple types of objects, using adjusted settings may produce undesirable effects. For example, if an image depicts both a face and a plant, using a high-sharpness setting might cause the facial skin to appear uneven, while using a low-sharpness setting might cause the leaves and flowers of the plant to appear blended together. To avoid this undesirable effect, such adjusted settings are likely to be used very conservatively in ISPs that apply tuning settings globally to all pixels.

[0057] Figure 2 Figure 200 is a conceptual diagram 200 illustrating image processing using a category graph 230 and a confidence graph 235. Figure 200 shows three hardware components of the image capture and processing device 100: an image sensor 205, an ISP 240, and a classification engine 220. The image sensor 205 can be... Figure 1An example of an image sensor 130. ISP 240 can be... Figure 1 An example of ISP 154. Classification engine 220 can be an example of the following: Figure 1 The host processor 152, Figure 1 ISP 154 Figure 1 The image processor 150, central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), another type of processor 1510, or some combination thereof.

[0058] Image sensor 205 receives light 202 from a scene being captured by image capture and processing device 100. In Figure 200, the scene being captured is a child eating food from a plate on a table. Image sensor 205 captures raw image data 210 based on the light 202 from the scene. Raw image data 210 consists of signals from photodiodes in the photodiode array of image sensor 205, which in some cases are amplified via an analog gain amplifier and in others are converted from analog to digital format using an analog-to-digital converter (ADC). Raw image data 210 typically includes separate image data from different photodiodes with different color filters. For example, if image sensor 205 uses Bayer color filters, the raw image data 210 includes image data corresponding to the red-filtered photodiode, image data corresponding to the green-filtered photodiode, and image data corresponding to the blue-filtered photodiode.

[0059] Image sensor 205 sends a first copy 210 of the original image data to ISP 240. Image sensor 205 sends a second copy 215 of the original image data to classification engine 220. In some cases, the second copy 215 of the original image data may be reduced, for example, at a ratio of 1:2, 1:4, 1:8, 1:16, 1:32, 1:64, another ratio higher than 1:64, or another ratio between any of the previously listed ratios. Although for simplicity, the second copy 215 of the original image data is shown as having been reduced in FIG. 200, it should be understood that classification engine 220 may reduce the second copy 215 of the original image data upon receiving it. In some cases, the second copy 215 of the original image data may be reduced before it is sent to and / or received by classification engine 220. For example, a second copy 215 of the original image data may be reduced at image sensor 205, at ISP 240, at a reduction unit (not shown) separate from image sensor 205 and ISP 240, or some combination thereof. Reduction techniques used to reduce the second copy 215 of the original image data may include nearest neighbor reduction, bilinear interpolation, bicubic interpolation, Sinc resampling, Lanczos resampling, box sampling, mipmapping, Fourier transform scaling, edge-oriented interpolation, high-quality scaling (hqx), or some combination thereof. In some cases, the second copy 215 of the original image data may be at least partially processed by ISP 240 and / or one or more additional components before being sent to and / or received by classification engine 220. For example, the second copy 215 of the original image data may be de-mosaiced by ISP 240 before being received by classification engine 220. Before the second copy 215 of the original image data is received by the classification engine 220, the second copy 215 of the original image data can be converted from one color space (e.g., RGB color space) to another color space (e.g., YUV color space). This process can be performed before or after reducing the size of the second copy 215 of the original image data.

[0060] In some examples, classification engine 220 receives a scaled-down second copy 215 of the original image data. In some examples, classification engine 220 receives the second copy 215 of the original image data and performs scaling to generate a scaled-down second copy 215 of the original image data. Classification engine 220 can use the scaled-down second copy 215 of the original image data to generate a category map 230 and a confidence map 235. For example, classification engine 220 can divide the scaled-down second copy 215 of the original image data into different image regions based on the detection of different object categories within different image regions in the scaled-down second copy 215 of the original image data. As an illustrative example, the category map 230 shown in Figure 200 includes two image regions corresponding to a child's face and arms, which are shaded with a first (diagonal stripe) shading pattern and labeled "skin," indicating that classification engine 220 detected skin in these image regions. Similarly, the category map 230 shown in Figure 200 includes image regions corresponding to hair on a child's head, which are shaded with a second (diagonal stripe) shading pattern and labeled "hair," indicating that the classification engine 220 has detected hair in that image region. In some cases, the classification engine 220 may also identify regions with eyelashes, eyebrows, beards, mustaches, and / or other hair objects as hair. Other image regions shown in category diagram 230 include: an image region where the classification engine 220 detects a shirt, which is shaded with a third (diagonal stripe) shading pattern and labeled "shirt"; several image regions where the classification engine 220 detects food, which are shaded with a fourth (diagonal stripe) shading pattern and labeled "food"; two image regions where the classification engine 220 detects fabric, which are shaded with a fifth (diagonal stripe) shading pattern and labeled "fabric"; three image regions where the classification engine 220 detects metal, which are shaded with a sixth (diagonal stripe) shading pattern and labeled "metal"; and two image regions where the classification engine 220 is unsure what they depict, which are shaded with a seventh (cross-shading) shading pattern and labeled "undefined". The image regions classified by the classification engine 220 as depicting different categories of objects can depict different objects, different types of objects, different materials, different substances, different elements, different components, objects with different properties, or some combination thereof. Figure 2Different shading patterns in category diagram 230 can represent different values ​​stored at corresponding pixel positions in category diagram 230, such as different colors, different gray shades, different numbers, different characters, or different bit sequences. In some cases, an image region determined to depict a specific object category can be referred to as an image region, object image region, image object region, category image region, image category region, category region, object category region, object category image region, image object category region, or a combination thereof. For example, an image region determined to depict a first object category can be referred to as a first object image region, an image region determined to depict a second object category can be referred to as a second object image region, and so on.

[0061] Confidence graph 235 identifies the level of confidence that classification engine 220 has regarding its classification of a given pixel in category graph 230. A region of image data with a specific confidence level may be referred to as a confidence region, confidence image region, image region, area, part, or a combination thereof. Pixels shown in white in confidence graph 235 represent high confidence levels, such as confidence levels exceeding a high threshold percentage (e.g., 90%). Pixels shown in black in confidence graph 235 represent low confidence levels, such as confidence levels falling below a threshold percentage (e.g., 10%). Confidence graph 235 also includes six different shades of gray (in addition to black and white), each representing a confidence level falling within a different confidence range between the high and low threshold percentages. For example, the lightest gray shade (still darker than white) can represent a confidence value between 90% and 80%, the next gray shade (one tone darker than the previous listed gray shade) can represent a confidence value between 80% and 70%, the next gray shade (one tone darker than the previous listed gray shade) can represent a confidence value between 70% and 60%, and so on. Including black, white, and six gray shades in between, example confidence map 235 includes a total of eight gray shades, corresponding to eight possible confidence levels. In some examples, the confidence level for a specific pixel can be stored as a 3-bit value. Classification engine 220 sends classification map 230 and confidence map 235 to ISP 240. Confidence map 235 can visually appear as having visible bands between different gray shades, such as gradients between the shades. In some examples, these gray shades can be mapped to confidence levels in the opposite direction to that described above, such that black and darker gray shades represent higher confidence values, while white and lighter gray shades represent lower confidence values.

[0062] In some cases, the category graph 230 and the confidence graph 235 may be a single file, a data stream, and / or a collection of metadata. Any discussion herein regarding either the category graph 230 or the confidence graph 235 should be understood to potentially include both. In one example, the single file may be an image. For example, each pixel of the image may include one or more values, which correspond to a classification and confidence level associated with a corresponding pixel in the second copy 215 of the original image data. In another example, the single file may be a matrix or a table, where each cell of the matrix or table stores a value corresponding to a classification and confidence level associated with a corresponding pixel in the second copy 215 of the original image data. For a given pixel in the second copy 215 of the original image data, the file stores the value in the corresponding cell or pixel. In some examples, a first plurality of bits in the stored values ​​represent the classification engine 220 classifying the pixel as the object category it depicts. In such an example, a second plurality of bits in the stored values ​​may represent the confidence level of the classification engine 220 in classifying the pixel as the object category it depicts.

[0063] In an illustrative example, the value stored in the file can be 8 bits in length, which can be referred to as a byte or octet. The first plurality of bits identifying the object category can be 5 bits of an 8-bit value, such as the earliest or most significant 5 bits of an 8-bit value. In the case of 5 bits, the first plurality of bits can identify 32 possible object categories. The first plurality of bits can represent the most significant bit (MSB) of the stored value. In the illustrative example above, the second plurality of bits representing the confidence level can be 3 bits of an 8-bit value, such as the last or least significant 3 bits of an 8-bit value. In the case of 3 bits, the second plurality of bits can identify 8 possible confidence levels. The second plurality of bits can represent the least significant bit (LSB) of the stored value. In some cases, the first plurality of bits may be later than the second plurality of bits within this value. In some cases, different subdivisions in terms of bit length are possible. For example, the first plurality of bits and the second plurality of bits can each include 4 bits. The first plurality of bits can include 1 bit, 2 bits, 3 bits, 4 bits, 5 bits, 6 bits, 7 bits, or 8 bits. The second set of bits can include 1 bit, 2 bits, 3 bits, 4 bits, 5 bits, 6 bits, 7 bits, or 8 bits. In some examples, the value stored in the file can be less than or greater than 8 bits in length.

[0064] In some examples, as described above, confidence graph 235 and category graph 230 are single files storing two separate values ​​per pixel of a second copy 215 of the original image data, where one of these values ​​represents the object category and the other represents the confidence level. In some examples, confidence graph 235 and category graph 230 are separate files, where one file stores values ​​representing the object category and the other file stores values ​​representing the confidence level.

[0065] In some cases, classification engine 220 enlarges the class map 230 and confidence map 235 from the size of a scaled-down second copy 215 of the original image data to the size of a first copy 215 of the original image data and / or to the size of the processed image 250. This enlargement process can use nearest neighbor (NN) magnification or dedicated class map magnification (CMUS), also known as NN magnification modified with spatial weighted filtering, as in at least Figure 9 , Figure 10 and Figure 11 As shown and discussed. In some cases, the classification engine 220 mixes or merges the category graph 230 and the confidence graph 235 into a single file before sending the single file to the ISP 240.

[0066] The ISP 240 receives a first copy 210 of the raw image data from the image sensor 205 and a category map 230 and a confidence map 235 from the classification engine 220. In some cases, while the classification engine 220 generates the category map 230 and the confidence map 235 and / or before the ISP 240 receives the category map 230 and the confidence map 235 from the classification engine 220, the ISP 240 may perform certain early processing tasks on the first copy 210 of the raw image data. These early image processing tasks may include, for example, demosaicing, color space conversion (e.g., from RGB to YUV), pixel interpolation, and / or downsampling. However, in other cases, the ISP 240 may delay some or all of these early image processing tasks until the ISP 240 receives the category map 230 and the confidence map 235 from the classification engine 220.

[0067] Once the ISP 240 receives the category map 230 and confidence map 235 from the classification engine 220, the ISP 240 uses the category map 230 and confidence map 235 from the classification engine 220 to process the image. The ISP 240 includes multiple modules that control the application of different ISP tuning parameters, which can be set to different settings. In one example, the ISP tuning parameters include noise reduction (NR), sharpening, tone mapping (TM), and color saturation (CS). In some cases, the ISP tuning parameters may also include additional parameters such as gamma, gain, brightness, shadows, edge enhancement, color correction (CC), color mapping (CM) (e.g., based on 2D lookup tables and / or 3D lookup tables), color shift, color enhancement, image combination for high dynamic range (HDR), effects processing (e.g., background replacement, bokeh effect), artificial noise (e.g., grain) adder, demosaic, edge-oriented magnification, other processing parameters discussed herein, or combinations thereof. Different settings for each ISP module can include default settings (also known as default ISP tuning settings), one or more settings that increase the ISP tuning parameters relative to the default settings, and one or more settings that decrease the ISP tuning parameters relative to the default settings. For example, for noise reduction (NR) ISP tuning parameters, available settings can include a default noise reduction level, one or more increased noise reduction levels that perform more noise reduction than the default level, and one or more decreased noise reduction levels that perform less noise reduction than the default level. In some cases, one or more parameters among the different ISP tuning parameters can include sub-parameters. Settings for such ISP tuning parameters can include values ​​or modifications to one or more of these sub-parameters. For example, NR ISP tuning parameters can include sub-parameters that include luma NR intensity, chroma NR intensity, and a temporal filter (e.g., for video or sequence denoising). Settings for NR can include modifications to the luma NR intensity sub-parameter, the chroma NR intensity sub-parameter, and / or the temporal filter sub-parameter. In some examples, instead of or in addition to color saturation (CS), the color saturation (CS) module 350 may control color correction (CC) and / or color mapping (CM) and / or color shift and / or color enhancement. The color saturation (CS) module 350 may be referred to as a module for any of the parameters listed above (e.g., CC module, CM module, color shift module, color enhancement module). In some cases, the color saturation (CS) module 350 may be referred to as a color processing module. In some examples, the ISP pipeline 305 may include separate color correction (CC) and / or color mapping (CM) and / or color shift and / or color enhancement modules. In some examples, noise reduction (NR) includes spatial noise reduction, temporal noise reduction, or both. While the NR module 320 is in Figure 3 While shown as a single module, in some examples, the ISP pipeline 305 may include separate spatial noise reduction modules and separate temporal noise reduction modules.

[0068] When the ISP 240 processes the first copy 210 of the original image data, it processes each image region differently based on the category map 230, according to which object category is depicted in each image region of the first copy 210 of the original image data. Specifically, if the category map 230 identifies the first image region as depicting skin, the ISP 240 processes the first image region according to settings corresponding to skin. Settings corresponding to skin can be stored as a specific modification to the default intensity level for applying a specific parameter (corresponding to skin). If the category map 230 identifies the second image region as depicting hair, the ISP 240 processes the second image region using settings corresponding to hair. Settings corresponding to hair can be stored as a specific deviation from the default intensity level for applying a specific parameter (corresponding to hair). The ISP 240 can identify the settings to use from a lookup table, database, or other data structure that maps object category identifiers in the category map 230 to corresponding settings.

[0069] In some cases, the ISP 240 can process different pixels within an image region differently based on the confidence level associated with each pixel in the confidence map 235. The ISP 240 can do this using a combination of the category map 230 and the confidence map 235. For example, in... Figure 3 The graph decoder component of the ISP 240 shown and discussed herein can use the category graph 230 and the confidence graph 235 to generate modification amounts, as discussed in relation to... Figure 4 As shown and discussed. Similarly, in some cases, in Figure 3 The smooth transition graph processor 365 shown and discussed here can use the category graph 230 and the confidence graph 235 to generate the modification amount, as shown in the discussion here. Figure 7 As shown and discussed, the ISP 240 can apply specific ISP tuning parameters, such as noise reduction, sharpening, tone mapping, and / or color saturation, to image data based on the amount of modification. Specifically, the settings for the ISP 240 to apply specific ISP tuning parameters to a given pixel of the image data depend on the amount of modification applied, as shown in... Figure 4 , Figure 5A , Figure 5B and Figure 5C As shown and discussed therein, the modification amount controls the ISP 240 to apply specific ISP tuning parameters to a given pixel of the image data based on the modification amount, determining the intensity or weight of that pixel.

[0070] Finally, the ISP 240 processes different image regions of the first copy 210 of the original image data using different modules configured for each image region based on category map 230 and / or confidence map 235, thereby generating a processed image 250. As previously described, the settings corresponding to the object image regions can be stored as specific deviations from the default intensity level for the application-specific parameters (corresponding to the object category). Although not shown in Figure 200, the ISP 240 can, for example, use I / O 156 and / or I / O 160 to send the processed image 250 to a storage device. Storage devices may include image buffers, random access memory (RAM) 140 / 1525, read-only memory (ROM) 145 / 1520, cache 1512, storage device 1530, secure digital (SD) cards, mini SD cards, micro SD cards, smart cards, integrated circuit (IC) memory cards, compressed optical discs, portable storage media, hard disk drives (HDDs), solid-state drives (SSDs), flash drives, non-transitory computer-readable storage media, any other type of memory or storage device discussed herein, or some combination thereof. Although not shown in Figure 200, ISP 240 may send the processed image 250 to a display buffer and / or a display screen or projector, such that the image is rendered and displayed on the display screen and / or using a projector.

[0071] In some cases, one or both of the first copy 210 and the second copy 215 of the original image data can simply be referred to as the original image data, or simply as the image data. For example... Figure 2 As shown, image sensor 205 can send a first copy 210 and a second copy 215 of the original image data to ISP 240 and / or classification engine 220. Alternatively, image sensor 205 can simply send a single copy of the original image data to a receiver component, which may be ISP 240 and / or classification engine 220 and / or... Figure 2 Another image processing component, not shown. This receiver component can generate one or more copies of the original image data, and sends one or more copies as a first copy 210 and / or a second copy 215 of the original image data to the ISP 240 and / or the classification engine 220. The receiver component can use and / or send the original copy 210 of the original image data it receives from the image sensor as a first copy 210 and / or a second copy 215 of the original image data.

[0072] In some cases, different image capture settings can be generated for different image regions of the image data, including settings for focus, exposure time, aperture size, ISO, flash, any combination thereof, and / or other image capture settings discussed herein. In some examples, one or more of these image capture settings can be determined at ISP 240. In some examples, one or more of these image capture settings can be sent back to image sensor 205 for application to the image data. In some cases, different image frames can be captured by image sensor 205 using different image capture settings, and then combined by ISP 240, host processor 152, image processor 150, another image processing component, or some combination thereof, such that different image regions are obtained from the image frames captured using different image capture settings. In some examples, different post-processing settings can also be generated for different image regions of the image, including settings for brightness, contrast, saturation, hue level, histogram, any combination thereof, and / or other processing settings discussed herein. In some cases, applying settings at the ISP 240 can allow for greater control over the resulting processed image and a higher quality of the applied processing (e.g., compared to post-processing settings). This is because the ISP 240 receives raw image data from the image sensor 205 as its input, while post-processing is typically applied to the image 250 already generated by the ISP 240.

[0073] In some cases, users can manually set at least one of the different settings for different object categories using the user interface, such as ISP tuning settings, image capture settings, and / or post-processing settings. For example, the user interface may receive input from the user specifying settings that indicate that increased sharpness settings should always be applied to image areas depicting text. Similarly, the user interface may receive input from the user specifying settings that indicate that decreased sharpness settings and increased noise reduction settings should always be applied to image areas depicting faces to make the skin of the face appear smoother. In some cases, at least one of the different settings for different object categories can be automatically set by: ISP 240, classification engine 220, host processor 152, image processor 150, application (e.g., an application for image post-processing), another image processing component, or some combination thereof. In some cases, at least one of the different settings for different object categories can be automatically set based on manually set settings, for example, by automatically determining settings that deviate from manually set settings based on modification amounts (such as multipliers, offsets, or logic-based modification amounts). The deviation from the modification amount can be predetermined or automatically determined, for example, based on the degree to which one object category is determined to differ from another object category regarding specific visual features (such as texture or color). For example, the ISP 240 can determine that similar or identical settings should be applied to image areas depicting line art as well as image areas depicting text. The ISP 240 can determine that the settings applied to image areas depicting skin with stubble should be approximately half the settings applied to image areas depicting skin as well as those applied to image areas depicting longer hair. The ISP 240 can determine that the settings applied to image areas depicting plaster walls should be similar to those applied to image areas depicting brick walls, but with a 10% increase in noise reduction.

[0074] Figure 3 This is a conceptual diagram 300 illustrating an ISP pipeline 305 for object-based classification image processing. The ISP pipeline 305 illustrates operations performed by components of the ISP 240. The operations and components of the ISP pipeline 305 are arranged in an exemplary layout and order as shown in the flowchart.

[0075] The inputs to ISP pipeline 305 and, consequently, to ISP 240 are shown on the left side of ISP pipeline 305. The inputs to ISP pipeline 305 include a category graph 230, a confidence graph 235, and a first copy 210 of the image data. The first copy 210 of the image data may be in a color filter domain (e.g., Bayer domain), RGB domain, YUV domain, or another color domain discussed herein. Although demosaicing and gamut transformation are not shown in Figure 300, it should be understood that ISP 240 may perform demosaicing and / or gamut transformation before, after, and / or between any two operations shown in Figure 200. Category graph 230 and confidence graph 235 are shown as being received twice by different elements of ISP pipeline 305. However, it should be understood that the ISP 240 can receive the category map 230 and the confidence map 235 at once, and internally distribute the category map 230 and the confidence map 235 to all appropriate components and elements of the ISP 240.

[0076] The ISP pipeline 305 receives the category map 230 and the confidence map 235 from the classification engine 220 and passes them through multiple graph decoders 325, 335, 345, and 355, each corresponding to a different module. Before passing the category map 230 and the confidence map 235 to the graph decoders 325, 335, 345, and 355, the ISP pipeline 305 can amplify the category map 230 and the confidence map 235 using an amplifier 310 (e.g., using nearest neighbor (NN) amplification and / or dedicated category map amplification (CMUS)). The amplifier 310 can amplify the category map 230 and the confidence map 235 such that the dimensions of the category map 230 and the confidence map 235 match the dimensions of the first copy 210 of the original image data and / or the dimensions of the processed image 250. In some cases, at least some of the amplification discussed with respect to the amplifier 310 may occur at the classification engine 220 before the ISP 240 receives the category map 230 and the confidence map 235. In some cases, the category graph 230 and the confidence graph 235 can be amplified once at the classification engine 220 and again at the amplifier 310 of the ISP 240.

[0077] Regardless of whether the ISP pipeline 305 uses amplifier 310 to amplify the category map 230 and confidence map 235, the ISP 240 receives the category map 230 and confidence map 235 and passes them to the image decoder 325 corresponding to the noise reduction (NR) module 320. Based on the category map 230 and confidence map 235, the image decoder 325 generates one or more modification amounts 327. The NR module 320 can use one or more modification amounts 237 to determine settings for NR to apply to different pixels of the first copy 210 of the original image data. The NR module 320 generates NR-processed image data by processing the first copy 210 of the original image data based on the modification amounts 327. The NR module 320 can send the NR-processed image data to the sharpening module 330. In some examples, NR includes spatial noise reduction, temporal noise reduction, or both. Although the NR module 320 in Figure 3 While shown as a single module, in some examples, the ISP pipeline 305 may include separate spatial noise reduction modules and separate temporal noise reduction modules.

[0078] Image decoder 325 passes category map 230 and confidence map 235 to image decoder 335 corresponding to sharpening module 330. Based on category map 230 and confidence map 235, image decoder 335 generates one or more modification amounts 337. Sharpening module 330 can use one or more modification amounts 337 to determine sharpening settings to apply to different pixels of the NR-processed image data from NR module 320. Sharpening module 330 processes the NR-processed image data based on modification amounts 337 to generate sharpened image data and sends the sharpened image data to tone mapping (TM) module 340.

[0079] Image decoder 335 passes category map 230 and confidence map 235 to image decoder 345 corresponding to TM module 340. Based on category map 230 and confidence map 235, image decoder 345 generates one or more modification amounts 347A. TM module 340 uses one or more modification amounts 347A to determine settings for TM to apply to different pixels of the sharpened image data from sharpening module 330. TM module 340 generates TM-processed image data by processing the sharpened image data based on modification amounts 347A and sends the sharpened image data to color saturation (CS) module 350.

[0080] Image decoder 345 passes category map 230 and confidence map 235 to image decoder 355 corresponding to CS module 350. In some cases, a delay 315 is applied between image decoder 345 and image decoder 355. Based on category map 230 and confidence map 235, image decoder 355 generates one or more modification amounts 357A, and CS module 350 uses one or more modification amounts 357A to determine settings for CS to apply to different pixels of TM-processed image data from TM module 340. CS module 350 generates CS-processed image data by processing the TM-processed image data based on modification amounts 357A. In some examples, delay 315 may be used to synchronize the reception of modification amounts 357A from image decoder 355 at CS module 350 with the reception of TM-processed image data at CS module 350. A similar delay can be inserted between any two elements of the image capture and processing device 100 (including the ISP 240) to help synchronize the transmission and / or reception of other signals within the image capture and processing device 100. In some cases, the CS-processed image data is then output by the ISP 240 as a processed image 250. In some cases, the ISP 240 performs one or more additional image processing operations on the CS-processed image data to generate the processed image 250. These one or more additional image processing operations may include, for example, reduction, enlargement, gamma adjustment, gain adjustment, another image processing operation discussed herein, or some combination thereof. In some cases, image processing using one of the ISP tuning parameter modules 320, 330, 340, or 350 can be skipped. In some examples, the image decoders 325, 335, 345, or 355 corresponding to the skipped ISP tuning parameter modules can be skipped. If one or both of these are skipped, a delay similar to delay 315 can be added in place of the skipped modules (ISP parameter modules and / or corresponding image decoders) to ensure that the processing elements move forward in a synchronized manner. In some cases, the image decoders 325, 335, 345 and / or 355 can internally track timing and detect when a module is skipped or removed, and can dynamically adjust the timing to generate modifications and / or send modifications to the corresponding ISP tuning parameter modules 320, 330, 340 or 350.

[0081] Delay 315 is a module agnostic to the category graph 230 and confidence graph 235, wherein it receives the category graph 230 and confidence graph 235 from the graph decoder 345 and sends the category graph 230 and confidence graph 235 to the next graph decoder 355 without generating any modifications using the category graph 230 and / or confidence graph 235. In some cases, other components agnostic to the category graph 230 and / or confidence graph 235 may be included (but not shown) within the ISP pipeline 305. In some cases, delay 315 may be removed. In some cases, one or more delay modules similar to delay 315 may be inserted between any two other components: ISP pipeline 305, ISP 240, any graph decoder, Smooth Transition Graph Processor (STMP) 365, classification engine 220, image capture and processing device 100, computing system 1500, any component of any of these modules, any other component or module or device discussed herein, or combinations thereof.

[0082] In some cases, ISP 240 can pass the category graph 230 and confidence graph 235 to the smooth transition graph processor (STMP) 365. ISP 240 can use STMP 365 to generate modification amounts and pass those modification amounts to at least some of the modules 320, 330, 340, and 355 (instead of or except for at least some of the graph decoders 325, 335, 345, and 355). STMP 365 can be used to create smooth transitions between them, at least... Figure 6 , Figure 7 and Figure 8 As shown and discussed. In some examples, before STMP 365 receives the category map 230 and confidence map 235, reducer 360 reduces the category map 230 and confidence map 235. In concept diagram 300, reducer 360 is shown as a component not part of ISP 240; therefore, ISP 240 can receive the category map 230 and confidence map 235 from classification engine 220, and the reduced version of the category map 230 and confidence map 235 from reducer 360. Reducer 360 can receive the confidence map 235 from classification engine 220. In some examples, reducer 360 may be part of ISP 240, such that ISP 240 receives class map 230 and confidence map 235, routes class map 230 and confidence map 235 to its own reducer 360, and then passes the reduced class map 230 and confidence map 235 to STMP 365.

[0083] In Figure 300, STMP 365 is shown generating an alternative modification 347B for TM module 340 and passing the alternative modification 347B to TM module 340 for use when processing sharpened image data. STMP 365 is also shown generating an alternative modification 357B for CS module 350 and passing the alternative modification 357B to CS module 350 for use when processing TM-processed image data. Although not shown in Figure 300, STMP 365 may also generate modifications for NR module 320 and / or sharpening module 325 and pass the generated modifications to NR module 320 and / or sharpening module 325.

[0084] Figure 300 illustrates the serial transmission of category map 230 and confidence map 235 from one to the next among graph decoders 325, 335, 345, and 355. However, ISP 240 can transmit copies of category map 230 and confidence map 235 in parallel to two or more of graph decoders 325, 335, 345, and 355. In this way, graph decoders 325, 335, 345, and 355 can generate modifications 327, 337, 347A, and 357A in parallel, potentially improving image processing efficiency.

[0085] Although STMP 365 and amplifier 310 are shown as components of ISP pipeline 305 and ISP 240, in some examples, at least one of them may be separate from ISP pipeline 305 and / or ISP 240. For example, at least one of STMP 365 and / or amplifier 310 may be part of classification engine 220, another component of image capture and processing device 100, another component of computing system 1500, or some combination thereof. Although reducer 360 is shown as a component separate from ISP pipeline 305 and ISP 240, in some examples, reducer 360 may be part of ISP pipeline 305 and / or ISP 240.

[0086] For illustrative purposes, the different ISP parameter modules are shown and described in a specific order. It should be understood that in alternative embodiments, the ISP parameter modules may be arranged in a different order than described, and / or the operations performed by the ISP parameter modules may be performed in a different order than described. For example, the TM module 340 may be located before the sharpening module 330 and / or the NR module 320.

[0087] Figure 4This is a conceptual diagram 400 illustrating the pipeline of graph decoder 325. Conceptual diagram 400 includes operations performed by components of graph decoder 325 of ISP 240. Graph decoder 325 corresponds to NR module 320, and Figure 400 illustrates the generation of modification amount 455 of modification amount 327 and the sending of modification amount 455 to NR module 320. The operations and components of graph decoder 325 are arranged in an exemplary layout and order as shown in the flowchart.

[0088] Image decoder 325 receives a category map 230 and a confidence map 235 generated by classification engine 220 and received by ISP 240. In some examples, image decoder 325 may include a delayed line buffer 410. Delayed line buffer 410 may delay sending the category map 230 and confidence map 235 from image decoder 325 to image decoder 335, which corresponds to the sharpening module 330 in the next line of ISP pipeline 305. Delayed line buffer 410 may also delay sending modification amount 327 (including modification amount 455) to NR module 320, such that the timing of NR module 320 receiving modification amount 327 is synchronized with the timing of NR module 320 receiving the first copy 210 of the original image data.

[0089] like Figure 4 As shown, the generator 430, based on the category modification amount 465, obtains the category map 230 (e.g., from the buffer of the delayed line buffer 410 or directly). An example category map 230 is shown at the bottom of concept map 400, which uses different numerical values ​​to label each of several differently colored image regions as corresponding to one of several object categories. Specifically, the category map 230 of concept map 400 includes a first image region shaded with a first shading pattern and marked with "0", which represents humans as an object category. Figure 4 Category diagram 230 includes: several secondary image regions depicting trees and grass, all shaded with a second shading pattern and marked with "1", representing plants as an object category. A third image region in category diagram 230, shaded with a third shading pattern and marked with "2", represents the sky as an object category. A fourth image region in category diagram 230, shaded with a fourth shading pattern and marked with "6", represents asphalt roads as an object category. Finally, a fifth image region in category diagram 230, shaded with a fifth shading pattern and marked with "9", represents vehicles as an object category. Figure 4 Different shading patterns in category diagram 230 can represent different values ​​stored at corresponding pixel positions in category diagram 230, such as different colors, different gray shades, different numbers, different characters, or different bit sequences.

[0090] To generate the category-based modification 465, generator 430 cross-references object categories in different image regions of category graph 230 with reference to data structure 480. Data structure 480 can be, for example, a lookup table, database, dictionary, list, array, array list, different data structures that can store associations between values, or some combination thereof. Data structure 480 stores predetermined settings suitable for each object category and the associated ISP tuning parameters. Because graph decoder 325 corresponds to NR module 320, data structure 480 stores predetermined settings suitable for each object category and NR. Different predetermined settings can essentially represent different intensities of NR to be applied. In some examples, different predetermined settings are expressed as values ​​based on an absolute scale. In some examples, different predetermined settings are expressed as values ​​relative to other values ​​(e.g., relative to values ​​in the default settings).

[0091] At the bottom of concept diagram 400, an exemplary category-based modification amount 465 generated by generator 430 is shown. The exemplary category-based modification amount 465 is shown in grayscale, where lighter shades represent predetermined settings associated with higher NR levels, and darker shades represent predetermined settings associated with lower NR levels. Based on category diagram 230, a total of four different grayscale shades are used. Therefore, image regions classified in category diagram 230 as depicting the sky (2) and vehicles (9) will be processed with higher NR levels, image regions classified in category diagram 230 as depicting humans (0) will be processed with medium NR levels, image regions classified in category diagram 230 as depicting asphalt roads (6) will be processed with low NR levels, and image regions classified in category diagram 230 as describing plants (1) will be processed with the lowest NR level.

[0092] Within the graph decoder 325, the confidence graph 235 is sent to the generator 435 for a hybrid update of the category-based modification amount 465. An example confidence graph 235 is shown at the bottom of the concept graph 400. As previously discussed, the confidence graph 235 is shown with eight grayscale shades, where lighter shades represent higher confidence levels or degrees of confidence, and darker shades represent lower confidence levels or degrees of confidence. Figure 4The confidence map 235 can visually appear as visible bands between different gray shades, such as gradients existing between the shades. The confidence map 235 instructs the classification engine 220 to generate a category map 230 with a generally high confidence level, where most of the portions with lower confidence levels are around the edges between different image regions representing different object categories. The generator 435 can identify appropriate adjustments to predetermined settings in the category-based modification amount 465 based on the different confidence levels. These adjustments can also be specified in the data structure 480.

[0093] At category-confidence blending operation 440, the category-based modification amount 465 generated by generator 430 is blended with the blend update for category-based modification amount 465 generated by generator 435. Category-confidence blending operation 440 produces a category-confidence blended modification amount 470. The modification amount value for a pixel can be reduced from the amount determined in the category-based modification amount 465 based on the confidence level for a particular pixel in confidence map 235. For example, a pixel with the highest confidence level in confidence map 235 can retain its modification amount value from the category-based modification amount 465. On the other hand, the modification amount value for a pixel with a low confidence level in confidence map 235 can be reduced from the modification amount value from the category-based modification amount 465, thereby reducing the intensity of the effect applied by the ISP tuning parameter module at that pixel. For example, as shown in concept diagram 400, when the ISP tuning parameter module is NR module 320, a weaker NR effect is applied to pixels classified by classification engine 220 with low confidence, rather than to pixels classified by classification engine 220 with high confidence. In some examples, a blending operation 465 is performed using no operation value. In some examples, the category-confidence blending operation 440 may be referred to as a category-confidence adjustment operation.

[0094] An exemplary category-confidence blended modification amount 470 is shown at the bottom of concept diagram 400. The category-confidence blended modification amount 470 is shown in grayscale. Similar to the exemplary category-based modification amount 465, lighter gray shades in the category-confidence blended modification amount 470 represent settings associated with higher NR levels, while darker shades represent settings associated with lower NR levels. Because the blended update based on confidence values ​​is blended with predetermined settings and thus adjusts the predetermined settings (e.g., the modification amount value in category-based modification amount 465), the category-confidence blended modification amount 470 may have portions corresponding to settings that do not match predetermined settings associated with any particular object category. For example, blending can be performed by adding, subtracting, or multiplying the value in the blended update for category-based modification amount 465 with the settings in category-based modification amount 465. This finer control over the ISP tuning parameter module allows the ISP 240 to apply its ISP tuning parameters, for example, in a less forceful manner to regions in which it is classified with low confidence. This reduces the risk of applying inappropriate settings to a portion of the image due to misclassification. For those portions of the image most likely to be misclassified (those with low confidence in the classification), the settings are toned down or otherwise adjusted to allow for a more conservative application of the ISP tuning parameters. Figure 4 The category-confidence blended modification amount 470 can visually appear as visible bands between different gray shades, such as gradients existing between shades. The bands in the category-confidence blended modification amount 470 can be from... Figure 4 The bars in confidence graph 235 are inherited from the different shadows or combinations thereof corresponding to different image regions based on category-based modification amount 465.

[0095] The category-confidence blended modification 470 is then passed through a low-pass filter 445 to produce a filtered modification 475. An example of the filtered modification 475 is shown at the bottom of concept diagram 400. The filtered modification 475 is similar to the category-confidence blended modification 470, but makes the transition between different settings smoother. For example, the boundaries between image regions in the category-confidence blended modification 470 may include stripes caused by different confidence levels from confidence graph 235, different image regions in category-based modification 465, or combinations thereof. In the filtered modification 475, intermediate values ​​are used to smooth the stripes, resulting in a blurring effect. This allows for smoother transitions between different settings applied using the module (here, NR) than would be the case with the category-confidence blended modification 470. The filtered modification 475 is amplified using amplifier 450 to generate the final modification 455. Amplifier 450 can perform this amplification using, for example, nearest neighbor (NN) amplification, bilinear interpolation, bicubic interpolation, Sinc resampling, Lanczos resampling, box sampling, mipmapping, Fourier transform scaling, edge-oriented interpolation, high-quality scaling (HQX), or some combination thereof. Image decoder 325 sends a final modification amount 455 to NR module logic 405 of NR module 320. The final modification amount 455 can be one of a set of one or more modifications 327. NR module logic 405 of NR module 320 applies NR to each pixel of a first copy 210 of the original image data based on the intensity of the final modification amount 455. The intensity can range from a minimum intensity within a predetermined range (represented by the darkest gray in the exemplary filtered modification amount 475) to a maximum intensity within a predetermined range (represented by white in the exemplary filtered modification amount 475). In some examples, low-pass filter 445 may include a Gaussian blur filter. In some examples, the low-pass filter 445 can be supplemented or replaced by another type of filter or blur effect, such as an average filter, box blur, lens blur, radial blur, motion blur, shape blur, smart blur, surface blur, or a combination thereof.

[0096] As described above, Figure 400 illustrates the modification amount 327 generated by the graph decoder 325 corresponding to the NR module 320. The same process can primarily be used by graph decoder 335 to generate the modification amount 337 for the sharpening module 330, by graph decoder 345 to generate the modification amount 347 for the TM module 340, and by graph decoder 355 to generate the modification amount 357 for the CS module 350. The main difference between the other graph decoders 335, 345, and 355 is that different data structures 480 are used, which store predetermined settings for the modules corresponding to these graph decoders. Alternatively, data structure 480 can store predetermined settings for multiple modules, for example, in different columns of a table; in this case, the same data structure 480 can be used, but different columns can be queried.

[0097] Figure 5A This is a conceptual diagram 510 illustrating the application of a modification amount 545A at the ISP module, where the modification amount 545A is applied as a multiplier. An internal signal 540A is received. Internal signal 540A indicates that the module applies the ISP tuning parameters to a portion of the image data using default settings or default intensity. A modification amount 545A (such as a modification to which a low-pass filter (LPF) 445 has already been applied) is received. The modification amount 545A identifies the value for each pixel of the image data. Filtered modification amounts 475 and / or final modification amounts 455 are examples of modification amount 545A. Similar to filtered modification amount 475, modification amount 545A can be stored as an image with different gray shading (different brightness values) at each pixel. Alternatively, modification amount 545A can be stored as a matrix, table, or other data structure, where each cell corresponds to each pixel of the image data. For a given pixel in the image data, the module acquires internal signal 540A (which will apply the default settings of the ISP tuning parameters) and multiplies internal signal 540A by the value in modification amount 545A corresponding to that pixel in the image data.

[0098] For example, internal signal 54A0 can indicate that a default setting or default intensity of 3 will be applied to a specific ISP parameter. Modification amount 545A can include a value of 1.6 corresponding to a given pixel in the image data, meaning that the ISP tuning parameter is applied at that pixel in the image data at 1.6 times the default setting indicated by internal signal 540, or 3 * 1.6 = 4.8. Modification amount 545A can include a value of 0.8 corresponding to a different pixel in the image data, meaning that the ISP tuning parameter is applied at that pixel in the image data at 0.8 times the default setting indicated by internal signal 540A, or 3 * 0.8 = 2.4. The modified internal signal 550A is the result of this multiplication and is therefore the intensity at which the module ultimately applies the ISP tuning parameter to a given portion of the image data. In some cases, the modified internal signal 550A can be expressed as a decimal value or a fraction. Alternatively, the modified internal signal 550A can be rounded to the nearest integer, or a lower or upper bound function can be applied to round it to the nearest integer that is less than or greater than the multiplication result, respectively.

[0099] In some cases, the modified internal signal 550A, generated by multiplying the default setting from internal signal 540A by the modification amount 545A, may be equivalent to a predetermined setting in a predetermined set corresponding to different object categories and / or confidence levels. In some cases, the modified internal signal 550A may be between two settings in the predetermined set, or may be outside the range represented by the predetermined set.

[0100] Figure 5B Conceptual diagram 520 illustrates the application of a modification amount 545B at the ISP module, where the modification amount 545B is applied as an offset. Conceptual diagram 520 includes an internal signal 540B and a modification amount 545B. The internal signal 540B can be the same as the internal signal 540A. The modification amount 545B can be the same as the modification amount 545A. However, in conceptual diagram 520, the value in the modification amount 545B for a given pixel is added to the value in the internal signal 540B to produce a modified internal signal 550B.

[0101] For example, internal signal 540B can indicate that a default setting or default intensity of 3 will be applied to a specific ISP parameter. Modification amount 545B can include a value of 1.6 corresponding to a given pixel in the image data, meaning that the ISP tuning parameter is applied at that pixel by adding the default setting indicated by internal signal 540B to 1.6, or 3 + 1.6 = 4.6. Modification amount 545 can include a value of -0.8 corresponding to a different pixel in the image data, meaning that the ISP tuning parameter is applied at that pixel by adding the default setting indicated by internal signal 540B to -0.8, or 3 - 0.8 = 2.2. The modified internal signal 550B is the result of this summation and is therefore the intensity at which the module ultimately applies the ISP tuning parameter to a given portion of the image data.

[0102] In some cases, the modified internal signal 550B, generated by summing the default setting from internal signal 540B with the modification amount 545B, can be equivalent to a predetermined set of predetermined settings corresponding to different object categories and / or confidence levels. In some cases, the modified internal signal 550B can be between two settings in the predetermined set of settings, or can be outside the range represented by the predetermined set of settings.

[0103] Figure 5C Conceptual diagram 530 illustrates the application of a modification amount 545C at the ISP module, which is applied using logic 555 based on parameter 560. Conceptual diagram 530 includes an internal signal 540C and a modification amount 545C. The internal signal 540C may be the same as internal signals 540A and / or 540B. The modification amount 545C may be the same as modification amounts 545A and / or 545B. However, in conceptual diagram 530, the value of the modification amount 545C for a given pixel represents a change from one predetermined setting to another.

[0104] For example, internal signal 540C may indicate that a default setting or default intensity of 3 will be applied to a specific ISP parameter. Modification amount 545C may include a value of 2 corresponding to a given pixel in the image data, meaning that an ISP tuning parameter will be applied at that pixel in the image data with an intensity selected from the list by choosing a second consecutive larger predetermined setting from the list. For example, if the predetermined setting list includes the set {1.5, 3, 4, 6, 8}, then the modified internal signal 550C is 6 because 6 is two values ​​higher than the default setting (3) in the list. Similarly, if modification amount 545C has a value of -1 corresponding to a different pixel in the image data, and the same predetermined setting list is used, then the modified internal signal 550C is 1.5 because 1.5 is one value lower than the default setting (3) in the list. The predetermined setting list may be taken from data structure 480, may each correspond to a different object category, and in some cases may be referred to as parameter 560. In some examples, each predetermined setting in the predetermined setting list may represent a different object category and / or a different confidence level. For example, if object categories and confidence levels are represented using 8 bits in category graph 230 and confidence graph 235, the list of predetermined settings can include 256 different predetermined settings. In some cases, the list of predetermined settings can include intermediate predetermined settings located between two other predetermined settings corresponding to a particular object category and confidence level. Using such intermediate predetermined settings can help create a smoother transition with fewer bands in the processed image 250 and can assist in implementing the smoothing produced by the low-pass filter 445. In some examples, instead of or in addition to the operations discussed above, logic 555 can include combinations of multiplication 510, offset 520, and / or other arithmetic operations. In some examples, instead of or in addition to the operations discussed above, logic 555 can mix the modification amount 545C with data from parameter 560. In some examples, instead of or in addition to the operations discussed above, logic 555 can include conditional programming (e.g., if…else), loops, and / or other programming logic. The determination of the modified internal signal 550C as discussed above can be referred to as the application of logic 555 based on parameter 560, the application of a logic engine that uses logic 555 based on parameter 560 to determine the modified internal signal 550C, or a combination thereof.

[0105] Figure 6 This is a conceptual diagram illustrating visual image artifacts introduced by anomalies during image segmentation into image regions during the generation of the category map. Specifically, Figure 6 It includes a first image 610, a second image 620, and a third image 630. The first image 610 is an image of two buildings and a tree against a blue sky background. Figure 6The first image 610 in the image represents the raw image data that has not yet been processed by the ISP240 based on the object category.

[0106] The second image 620 is similar to the first image 610, but includes a white image region labeled as the sky image region 625. The sky image region 625 represents the portion within the first image 610 that the classification engine 220 detects as depicting the sky. In this example, the sky is an object category. Therefore, the sky image region 625 is the image region detected by the classification engine 220 as depicting the "sky" object category. The boundaries of the sky image region 625 include artifacts 640 in certain areas near the boundaries of the areas depicting the sky and the areas depicting buildings and trees in the first image 610. These anomalies may arise from imperfect detection of object categories, for example, due to similar blue shadows appearing in the sky and building windows, or due to complex boundaries of leaves.

[0107] The third image 630 represents a version of the first image 610 as follows: this version was processed by the ISP 240 based on object categories using a category map that includes the sky image region 625, and is based on the category division of the second image 620. At the location 620 of the artifact 640 in the sky image region 625 of the second image 620, the third image 630 includes visual image artifacts 645 with hue and color transitions. For example, in the third image 630, the sky regions near the boundaries of buildings and the sky, and near the boundaries of trees and the sky, appear brighter and less saturated than the rest of the sky. These artifacts 645, which ultimately become lighter and less saturated due to artifact 640, do not belong to the sky image region 625 of the second image 620. In similar situations, abrupt transitions from one setting to another can produce these types of visual image artifacts or similar visual image artifacts. In similar situations, smoother transitions between one setting and another can reduce the occurrence of such artifacts. A smoother transition can be achieved by generating a smoother amount of modification corresponding to the sky image region 625, for example by using... Figure 7 The techniques shown are used to generate smoothed modification amounts (such as in...). Figure 8 The amount of modification shown.

[0108] Figure 7This is a conceptual diagram 700 illustrating the pipeline of a Smooth Transition Map Processor (STMP) 365. The STMP 365 produces smooth transitions from one setting to another within the same image. As discussed below, the STMP 365 operates similarly to two map decoders 435 combined into a single component, providing a modification amount 755A to the TM module 340 and a modification amount 755B to the CS module 350. However, before generating the modification amounts 755A and 755B at the STMP 365, the category map 230 and confidence map 235 can be passed through an additional reducer 360 and a front-end (FE) 705. The additional reduction provided by the reducer 360 effectively creates a blurring effect at the boundaries between different settings when amplified by amplifiers 750A and 750B. In some cases, the STMP 365 can also use low-pass filters 745A and 745B, which are stronger than the low-pass filter 445, further smoothing the transition between different settings to reduce banding in the transition. In some examples, low-pass filters 745A and 745B may include Gaussian blur filters. In some examples, low-pass filters 745A and 745B may be supplemented or replaced by another type of filter or blur effect, such as average filter, box blur, lens blur, radial blur, motion blur, shape blur, smart blur, surface blur, or combinations thereof.

[0109] Similar to the graph decoder 435 STMP 365, the row buffer 710 routes the category graph 230 to two generators 730A and 730B for category-based modifications 765A and 765B, each operating similarly to generator 430 for category-based modifications 465. The row buffer 710 routes the confidence graph 235 to two generators 735A and 735B for mixed updates of category-based modifications 765A and 765B, each operating similarly to generator 435 for mixed updates of category-based modifications 465. Similar to the category-confidence mixing operation 440, the category-confidence mixing operation 740A mixes category-based modifications 765A with mixed updates for category-based modifications 765A. The resulting mixed modification is filtered by a low-pass filter 745A and amplified by an amplifier 750A to produce a modification 755A with a smooth transition (an example of modification 347B). The modification 755A with a smooth transition is sent from STMP 365 to TM module 340 in place of modification 347A. Similarly, similar to category-confidence mixing operation 440, category-confidence mixing operation 740B mixes the category-based modification 765B with a mixed update for the category-based modification 765B. The resulting mixed modification is filtered by a low-pass filter 745B and amplified by an amplifier 750B to produce a modification 755B with a smooth transition (an example of modification 357B). The modification 755B with a smooth transition is sent from STMP 365 to CS module 350 as an example of modification 347B.

[0110] The TM module 340 processes the sharpened image data 770A based on a modification amount 755A with a smooth transition to generate TM-processed image data 770B, which is then sent to the CS module 350. The CS module 350 processes the TM-processed image data 770B based on the modification amount 755B with a smooth transition to generate CS-processed image data 770C. The CS-processed image data 770C can be the processed image 250, or it can be sent to another component in the ISP 240 for further processing to produce the processed image 250. Although not shown in Figure 700, the STMP 365 can also generate modification amounts for the NR module 320 and / or the sharpening module 325, and pass the generated modification amounts to the NR module 320 and / or the sharpening module 325. In some examples, the TM module 340 can use the STMP-based modification 347B / 755A and the graph decoder-based modification 347A in parallel, enjoying both low-resolution processing modification and high-resolution processing modification. In some examples, the CS module 350 can use the STMP-based modification 357B / 755B and the graph decoder-based modification 357A in parallel, enjoying both low-resolution processing modification and high-resolution processing modification.

[0111] Figure 8 This is a conceptual diagram illustrating the use of a smooth transition map processor to smooth modifications corresponding to image regions. Specifically, Figure 8 It shows the basis Figure 6 Four versions of the modification amount were generated for the sky image region 625, and these modifications were generated by STMP 365 using different scaling. The first modification amount 810 was generated using 1:1 scaling, which means that the reducer 360 omits or does not perform reduction on the category map 230 and / or confidence map 235. Therefore, the boundary between the region corresponding to the sky image region 625 in the first modification amount 810 and other regions in the first modification amount 810 is clear.

[0112] The second modification 820 is generated using a 1:4 scaling ratio, meaning that the reducer 360 shrinks the category map 230 and / or confidence map 235 to one-quarter of its original size. Therefore, the boundary between the region corresponding to the sky image region 625 in the second modification 820 and other regions in the second modification 820 is more blurred compared to the same boundary in the first modification 810. Similarly, the reducer 360 uses a 1:16 scaling ratio to generate the third modification 830, thus reducing the category map 230 and / or confidence map 235 to one-sixth of its original size. Therefore, the boundary in the third modification 830 is more blurred compared to the boundary in the second modification 820. Finally, the reducer 360 uses a 1:64 scaling ratio to generate the fourth modification 840, thus reducing the category map 230 and / or confidence map 235 to one-sixty-fourth of its original size. Therefore, the boundary in the fourth modification 840 is more blurred compared to the boundary in the third modification 830. Figure 8 Compared to the scaling shown, even higher scaling levels are possible, such as 1:256 scaling. Scaling levels between any of the scaling levels discussed above are also possible, such as 1:3 scaling, 1:6 scaling, 1:10 scaling, 1:32 scaling, 1:50 scaling, 1:100 scaling, or 1:128 scaling.

[0113] Figure 9 Figure 900 illustrates the Category Map Amplifier (CMUS) 905 pipeline. The Category Map 230 typically cannot be amplified using interpolation-based amplification (such as bilinear or bicubic interpolation) because all values ​​in the Category Map 230 represent a specific object category. Interpolation can create intermediate values ​​that may reference unexpected or non-existent object categories. For example, the Category Map may contain a pixel with a value of 2 adjacent to a pixel with a value of 4. For instance, a value of 2 could represent the Sky object category, while a value of 4 could represent the Plant object category. Amplification using interpolation might create a pixel with a value of 3 between the pixel with a value of 2 and the pixel with a value of 4. A value of 3 might represent a different object category (such as Fabric), which might correspond to a completely different ISP tuning setting than Sky or Plant, and this could lead to visual artifacts if used for object-based classification image processing. Alternatively, a value of 3 might not represent any known object category at all, which could lead to errors or visual artifacts in the case of object-based classification image processing.

[0114] One approach that can be used to magnify a class map without the problems associated with interpolation-based magnification is Nearest Neighbor (NN) magnification. NN magnification does not create intermediate values. However, NN magnification can produce sharp, blocky edges. Sometimes, objects depicted with narrow curves in an image (such as a person's eyebrows, shadows, or clothing bands) can appear particularly blocky and inaccurate due to NN magnification. Class Map Magnification (CMUS) (also known as NN modified with spatial weighted filtering) magnifies the class map more accurately without introducing problems associated with interpolation, such as intermediate values. The improvement in magnification is particularly noticeable at boundaries between image regions and in narrow image regions.

[0115] The CMUS 905 pipeline in concept diagram 900 uses category diagram 230 as input to filter and magnification operation 920 and filter size determination operation 910. Filter size determination operation 910 can adaptively select a filter size from a set of filter sizes for magnifying a given pixel, such as filters with sizes of 2x2 pixels, 4x2 pixels, 2x4 pixels, or 4x4 pixels. To preserve finer details, smaller filters are used for narrow or small image regions in category diagram 230. Neighbor weights 915 based on confidence diagram 235 are provided to filter and magnification operation 920. Two examples are shown with double (x2) magnification 970 and quadruple (x4) magnification 975, where circular points (pixels) are interpolated from square points (neighboring pixels) based on the confidence of the square point's category, the distance from the square point to the circular point, or a combination thereof.

[0116] In operation 925, the cumulative weight for each category is calculated. An example of this calculation is provided in example 965. A neighboring pixel with a lower confidence level in confidence map 235 can have a lower weight in neighbor weight 915 compared to a neighboring pixel with a higher confidence level in confidence map 235, and therefore can contribute less to the total weight. Similarly, a neighboring pixel farther from the magnified pixel can also have a lower weight compared to a neighboring pixel closer to it, and therefore can contribute less to the total weight. In other words, a neighboring pixel closer to the magnified pixel can have a higher weight compared to a neighboring pixel farther from it. In operation 930, the category with the highest weight is used for the magnified pixel in the magnified category map 950. In some examples, amplifier 310 may use NN magnification, CMUS magnification, or some combination thereof to perform magnification. In some examples, one or more of amplifiers 450, 750A, and 750B may perform amplification using NN amplification, CMUS amplification, another amplification technique discussed herein, or some combination thereof.

[0117] Figure 10This is a graph showing a comparison between a class map magnified using nearest neighbor magnification and the same class map magnified using nearest neighbor magnification modified with spatial weighted filtering (which is applied using a class map amplifier (CMUS) 905). Figure 10 The category charts are based on the same images: the surface of a table, a photograph of a man, a photograph of a woman, three pens, and a corner of a tablet device placed on the table. These category charts all include various image areas represented using different shades of gray. Figure 10 The first category image 1010 in the image is scaled using NN magnification. Due to the use of NN magnification, the boundaries of the individual image regions in the first category image 1010 are extremely blocky and jagged.

[0118] Figure 10 The second category map 1020 is the same category map as the first category map 1010, but it is magnified using Category Map Magnification (CMUS) instead of ordinary nearest neighbor magnification. Category Map Magnification (CMUS) can also be referred to as NN magnification modified with spatial weighted filtering. Therefore, where appropriate, the boundaries between different image regions are more rounded and less blocky overall. The improvement in magnification fidelity is particularly noticeable in narrow image regions, such as the image region representing the belt of a woman's clothing in a photograph.

[0119] Figure 11 This is a conceptual diagram 1100 illustrating an example resolution of image data corresponding to a category map during zoom-out and zoom-up operations. In conceptual diagram 1100, raw image data 1105 captured by image sensor 205 has a 4K resolution plus electronic image stabilization (EIS) margin, resulting in a resolution of 4800x2700. At operation 1110, the raw image data is zoomed out to a resolution of 840x480. This is in... Figure 2 The reduction shown in the second copy 215 of the original image data can be performed using a reducer. This reducer can be part of the image sensor 205, located in the classification engine 220, ISP 240, or... Figure 2At another component not shown in the diagram, classification engine 220 divides the scaled-down image data from operation 1110 into image regions at operation 1115 to produce category map 1120, which also has a resolution of 848x480. In the example of concept diagram 1100, category map 1120 is scaled up once using NN scaling up operation 1125 to achieve a resolution of 1200x675. The scaled-up category map is then scaled up again to a resolution of 1920x1080 using CMUS scaling up operation 1130. In some examples, amplifier 310 may perform one or both of scaling up operations 1125 and / or 1130. In some examples, amplifier in classification engine 220 may perform one or both of scaling up operations 1125 and / or 1130. In some examples, amplifier in ISP 240 may perform one or both of scaling up operations 1125 and / or 1130.

[0120] Figure 12A This is a flowchart 1200 illustrating image processing techniques. The image processing techniques shown in flowchart 1200 can be performed by a device. This device may be an image capture and processing device 100, an image capture device 105A, an image processing device 105B, a classification engine 220, an ISP 240, an image sensor 205, one or more web servers for cloud services, a computing system 1500, or some combination thereof.

[0121] At operation 1205, as part of image processing techniques, the device receives image data captured by image sensor 205. In some cases, the device may include a connector coupled to image sensor 205, and this connector may be used to receive image data. The connector may include ports, jacks, wires, input / output (IO) pins, conductive traces on a printed circuit board (PCB), any other type of connector discussed herein, or some combination thereof. In some cases, the device may include image sensor 205.

[0122] In some examples, the image data may be raw image data. In some examples, the device may demosaic the image data. In one illustrative example, the device may demosaic the image data after receiving the image data in operation 1205 but before at least one of the other operations 1210-1235. In some examples, the device may convert the image data from a first color space to a second color space. In one illustrative example, the device may convert the image data from a first color space to a second color space after receiving the image data in operation 1205 but before at least one of the other operations 1210-1235. In some examples, the second color space is the YUV color space. In some examples, the second color space is the RGB color space. In some examples, the first color space is the RGB color space. In some examples, the first color space is the Bayer color space or another color space associated with one or more color filters on the image sensor 205.

[0123] At operation 1210, as part of image processing techniques, the device determines that a first object image region in the image data depicts a first object category among a plurality of object categories. At operation 1215, as part of image processing techniques, the device determines that a second object image region in the image data depicts a second object category among a plurality of object categories.

[0124] At operation 1220, as part of image processing techniques, the device generates a category map 230 by dividing image data into multiple object image regions, including a first object image region and a second object image region. Each of the multiple regions corresponds to one of multiple object categories (e.g., a first region corresponding to a first object, a second region corresponding to a second object, etc.). In some aspects, the device also generates a scaled-down copy of the image data by reducing its size. Generating a category map based on the image data includes generating a category map based on the scaled-down copy of the image data.

[0125] Although not shown in flowchart 1200, operation 1220 may also include generating a confidence map 235 based on image data. Confidence map 235 identifies multiple confidence levels corresponding to multiple portions of the image data. Each of the multiple confidence levels identifies the degree of confidence in determining that the corresponding portion of the multiple portions depicts one of multiple object categories. In one example, confidence map 235 and category map 230 are a single file storing a single value for each pixel. The first plurality of bits in this value represent the classification engine 220 classifying the pixel as the depicted object category. The second plurality of bits in this value represent the confidence level of the classification engine 220 in classifying the pixel as the depicted object category. In another example, confidence map 235 and category map 230 are a single file storing two values ​​for each pixel, where one value represents the object category and the other value represents the confidence level.

[0126] In some aspects, the device also magnifies the category map. In some examples, magnifying the category map may include magnifying the category map to a size that matches the size of at least one of the image data and the image. In some examples, the category map may be magnified using nearest neighbor magnification or using modified nearest neighbor magnification (e.g., modified using spatial weighted filtering, which may also be referred to herein as category map magnification (CMUS)). Magnifying the category map using nearest neighbor magnification modified with spatial weighted filtering may include: identifying a first filter size corresponding to a first object image region and a second filter size corresponding to a second object image region. The first filter size is smaller than the second filter size. Magnifying the category map using nearest neighbor magnification modified with spatial weighted filtering may include: magnifying a first pixel within the first object image region based on the first filter size and one or more weights associated with one or more confidence values ​​from a confidence map, the one or more confidence values ​​corresponding to one or more pixels adjacent to the first pixel. Using nearest-neighbor magnification modified by spatial weighted filtering to magnify a category map may include: magnifying a second pixel within a second object image region based on a second filter size and one or more weights associated with one or more confidence values ​​from a confidence map, wherein the one or more confidence values ​​correspond to one or more pixels adjacent to the second pixel.

[0127] At operation 1225, as part of the image processing technique, the device identifies a first object category corresponding to a first tuning setting for the image signal processor (ISP). At operation 1230, the image processing technique includes identifying a second object category corresponding to a second tuning setting for the ISP. The first and second tuning settings may include indicators of different intensities of applying noise reduction (NR) ISP tuning parameters during image data processing. The first and second tuning settings may include indicators of different intensities of applying sharpening ISP tuning parameters during image data processing. The first and second tuning settings may include indicators of different intensities of applying color saturation (CS) ISP tuning parameters during image data processing. The first and second tuning settings may include indicators of different intensities of applying tone mapping (TM) ISP tuning parameters during image data processing. The first and second tuning settings may include indicators of different intensities of applying gamma ISP tuning parameters during image data processing. The first and second tuning settings may include indicators of different intensities of applying different ISP tuning parameters during image data processing. Different ISP tuning parameters may include, for example, gain, brightness, shadow, edge enhancement, image combination for high dynamic range (HDR), special effects processing (e.g., background replacement, bokeh effect), artificial noise adder, demosaic, edge-oriented magnification, other processing parameters discussed herein, or combinations thereof.

[0128] In some cases, as described above, the first setting, the second setting, and / or the first and second tuning settings are defined based on user input associated with the first object image region and the second object image region. In some cases, as described above, at least one of the first setting and the second setting is determined automatically.

[0129] At operation 1235, as part of the image processing technique, the device generates an image by processing image data using an ISP tuned based on a category map. For example, the ISP may use a first tuning setting to process a first object image region in the image data. The ISP may use a second tuning setting to process a second object image region in the image data. In some cases, generating the image involves processing the raw image data using an ISP tuned based on both a category map and a confidence map.

[0130] In some aspects, the device also generates one or more modification amounts based on a category map. The one or more modification amounts identify at least one of a first bias or a second bias. The first bias is a bias applied by the ISP to a first object image region compared to a default setting during image data processing. The second bias is a bias applied by the ISP to a second object image region compared to a default ISP tuning setting during image data processing. In some aspects, the ISP identifies at least one of the first bias or the second bias by performing an arithmetic function of one or more modification amounts and a default tuning setting. The arithmetic function may include at least one of a multiplication function, an addition function, a subtraction function, a division function, or some combination thereof. The multiplication function may multiply one or more modification amounts by the default tuning setting, for example, in... Figure 5A As shown in the diagram. The summation function can add one or more modifications to the default tuning setting, for example, in... Figure 5C As shown. The subtraction function can subtract one or more modifications from the default tuning settings and vice versa. The division function can divide the default tuning settings by one or more modifications and vice versa. In some aspects, such as... Figure 5C As shown, the ISP identifies at least one of the first and second deviations based on an increment from a predetermined list of possible settings, including the default settings, where the increment is based on the amount of modification. In some aspects, the device can reduce the category map before generating one or more modifications based on the category map.

[0131] The device can generate one or more mixed modifications by mixing one or more modifications with information corresponding to a confidence map. The device can generate one or more filtered modifications by filtering one or more mixed modifications using a low-pass filter. The device can generate one or more amplified modifications by amplifying one or more filtered modifications. Using a category map-tuned ISP to process image data in operation 1235 can include processing the image data using one or more modifications, one or more mixed modifications, one or more filtered modifications, one or more amplified modifications, or some combination thereof.

[0132] The image processing techniques shown in flowchart 1200 may also include any operations shown or discussed in any of the figures in flowcharts 1250, 1300 and / or 1400.

[0133] Figure 12BThis is a flowchart 1250 illustrating image processing techniques. The image processing techniques shown in flowchart 1250 can be performed by a device. This device may be an image capture and processing device 100, an image capture device 105A, an image processing device 105B, a classification engine 220, an ISP 240, an image sensor 205, one or more web servers for cloud services, a computing system 1500, or some combination thereof.

[0134] At operation 1255, as part of image processing techniques, the device receives image data captured by image sensor 205. Operation 1205 of flowchart 1200 can be an example of operation 1255 of flowchart 1250.

[0135] At operation 1250, as part of image processing techniques, the device determines a first object image region in the image data that depicts a first object category among a plurality of object categories. Operation 1210 of flowchart 1200 can be an example of operation 1260 of flowchart 1250.

[0136] In operation 1265, as part of image processing techniques, the device determines a second object image region in the image data that depicts a second object category among multiple object categories. Operation 1215 of flowchart 1200 can be an example of operation 1265 of flowchart 1250.

[0137] At operation 1270, as part of image processing techniques, the device identifies multiple confidence levels corresponding to multiple confidence image regions in the image data, wherein each of the multiple confidence levels identifies a confidence level that describes one of multiple object categories with respect to the corresponding confidence image region among the multiple confidence image regions. Operation 1220 of flowchart 1200 may include operation 1270 of flowchart 1250.

[0138] At operation 1275, as part of image processing technology, the device uses an image capture process to generate an image based on image data. This includes applying different settings for the image capture process to different portions of the image data, which are identified based on a first object image region, a second object image region, and multiple confidence level image regions. In some examples, operation 1275 of flowchart 1250 may include at least a subset of at least one of operations 1220, 1225, 1230, and / or 1235 of flowchart 1200. For example, some portions of the different portions of the image data may be different portions of the first object image region with different confidence levels. Some portions of the different portions of the image data may be different portions of the second object image region with different confidence levels. Some portions of the different portions of the image data may be outside the first object image region and / or the second object image region.

[0139] In some examples, the image capture process includes generating one or more modification quantities. The one or more modification quantities may identify a first deviation from the default settings for the image capture process for a first object image region, a second deviation from the default settings for the image capture process for a second object image region, or both. Different settings for the image capture process may be based on one or more modification quantities. The default settings may be the default intensity of a specific parameter (e.g., an ISP parameter) being applied, and each deviation corresponding to each modification quantity may represent a reduction or strengthening of that default intensity. In some examples, as part of an image processing technique, the device adjusts one or more modification quantities. Adjusting one or more modification quantities may include combining one or more modification quantities with a mixed update based on multiple confidence levels corresponding to multiple confidence image regions (e.g., by...). Figure 4 The generator 435 generates a blended update, which is then blended with the blended update. Blending one or more modifications with the blended update can adjust at least one of a first bias and a second bias in at least one region of the image data. The modifications can further reduce or increase the intensity of the applied specific parameter (e.g., the ISP parameter).

[0140] In some examples, as part of image processing techniques, the device generates a category map that divides image data into multiple object image regions, including a first object image region and a second object image region. Each of the multiple object image regions corresponds to one of multiple object categories. The device can identify that the first object category corresponds to a first setting for the image capture process. The device can identify that the second object category corresponds to a second setting for the image capture process. In some examples, as part of image processing techniques, the device generates a confidence map that divides image data into multiple confidence image regions corresponding to multiple confidence levels. Different portions of the image data can be identified (e.g., by the device) based on the category map and the confidence map.

[0141] In some examples, the image capture process may include the processing of image data as described in operation 1235. In some examples, the first setting for the image capture process may be the first tuning setting discussed with respect to operations 1225 and 1235. In some examples, the second setting for the image capture process may be the second tuning setting discussed with respect to operations 1230 and 1235.

[0142] In some examples, the image capture process includes processing image data using an image signal processor (ISP). Different settings for the image capture process can be different tuning settings for the ISP. In some examples, different tuning settings for the ISP include applying different intensities of ISP tuning parameters during image data processing using the ISP. For example, ISP tuning parameters can be one of noise reduction, sharpening, color saturation, color mapping, color processing, and tone mapping. In some examples, different settings include settings associated with at least one of the following: lens position, flash, focus, exposure, white balance, aperture size, shutter speed, ISO, analog gain, digital gain, noise reduction, sharpening, tone mapping, color saturation, de-mosaic, color space conversion, shadows, edge enhancement, image combination for high dynamic range (HDR), effects, artificial noise addition, edge-oriented magnification, magnification, reduction, electronic image stabilization, or combinations thereof. In some examples, the device processes image data. Processing image data may include demosaicing the image data and / or converting the image data from a first color space to a second color space (e.g., between the Bayer color space, the RGB color space, and / or the YUV color space).

[0143] In some examples, as part of image processing techniques, the device receives user input associated with at least one of a first object image region and a second object image region. At least one of the different settings may be defined based on the user input and may correspond to either the first object image region or the second object image region. In some examples, applying different settings for the image capture process to different portions of the image data includes using an image signal processor (ISP) to apply different settings for the image capture process to different portions of the image data. In some examples, identifying the first object image region and the second object image region includes using a classification engine to identify the first object image region and the second object image region, the classification engine being at least partially located on an integrated circuit (IC) chip (such as an application-specific integrated circuit (ASIC) chip). In some examples, as part of image processing techniques, the device displays an image on a display.

[0144] The image processing techniques shown in flowchart 1250 may also include any operations shown or discussed in any of the figures in flowcharts 1200, 1300 and / or 1400.

[0145] Figure 13 This is a flowchart 1300 illustrating a transition smoothing technique. The image processing technique shown in flowchart 1300 can be performed by a device. This device can be an image capture and processing device 100, an image capture device 105A, an image processing device 105B, a classification engine 220, an ISP 240, an image sensor 205, one or more web servers for cloud services, a computing system 1500, or some combination thereof.

[0146] At operation 1305, the transition smoothing technique includes receiving a class map and a confidence map. At operation 1310, the transition smoothing technique includes reducing the class map. In some examples, operation 1310 can be skipped so that the class map is not reduced.

[0147] At operation 1315, the transition smoothing technique includes generating one or more modification amounts based on a category map. For example, the one or more modification amounts identify at least one of a first deviation from the default setting applied by the ISP in a first image region during image data processing and a second deviation from the default setting applied by the ISP in a second image region during image data processing. Figure 5A , Figure 5B and Figure 5C The internal signal 540 shown can represent an example of the default setting. Category-based modifications 465, 765A, and 765B can represent examples of one or more modifications based on the category graph. Generators 430, 730A, and 730B can perform operation 1315.

[0148] At operation 1320, the transition smoothing technique includes generating one or more mixed modification amounts by mixing one or more modification amounts with information corresponding to a confidence graph. Category-confidence mixing operations 440, 740A, and 740B (e.g., using confidence as a mixing factor, such as mixing modification amounts with no-operation equivalent modification amount values ​​based on confidence) can represent examples of operation 1320. Mixed updates for category-based modification amounts 465, 765A, and 765B generated by generators 435, 735A, and 735B can represent examples of information corresponding to a confidence graph.

[0149] At operation 1325, the transition smoothing technique involves generating one or more filtered modifications by filtering one or more mixed modifications using a low-pass filter (LPF). LPFs 445, 745A, and 745B can represent examples of LPFs for operation 1325.

[0150] At operation 1330, the transition smoothing technique includes generating one or more amplified modifications by amplifying one or more filtered modifications. Amplifiers 450, 750A, and 750B can perform operation 1330.

[0151] At operation 1335, the transition smoothing technique includes processing the image data using an ISP tuned based on one or more amplified modification amounts. In one example, operation 1335 may be performed by module logic of an ISP tuning parameter module (such as NR module logic 405 of NR module 320).

[0152] In some cases, one or more operations 1305-1335 of flowchart 1300 can be performed by the device that performs one or more operations 1205-1235 of flowchart 1200. In some cases, Figure 13 Transition smoothing technology can be Figure 12A It is part of the image processing technology. Figure 12A The image processing techniques can represent at least some of the operations of the classification engine 220 and / or ISP 240. Figure 13 The transition smoothing technique can represent at least some of the operations in the operation of STMP 365 and / or reducer 360.

[0153] The transition smoothing technique shown in flowchart 1300 may also include any operation shown or discussed in any of the figures in flowcharts 1200, 1250 and / or 1400.

[0154] Figure 14This is a flowchart 1400 illustrating image magnification technology. The image processing technology shown in flowchart 1400 can be performed by a device. This device may be an image capture and processing device 100, an image capture device 105A, an image processing device 105B, a classification engine 220, an ISP 240, an image sensor 205, one or more web servers for cloud services, a computing system 1500, or some combination thereof.

[0155] At operation 1405, the image magnification technique includes receiving a category map 230 and a confidence map 235. In some cases, the category map 230 and the confidence map 235 may be a single file containing the category information and confidence information for each pixel discussed above.

[0156] At operation 1410, the image magnification technique includes: identifying a first image region and a second image region of category figure 230, wherein the first image region is narrower than the second image region.

[0157] At operation 1415, the image magnification technique includes: identifying a first filter size corresponding to a first image region and a second filter size corresponding to a second image region, wherein the first filter size is smaller than the second filter size.

[0158] At operation 1420, the image magnification technique includes: magnifying a first pixel within a first image region based on a first filter size and one or more weights associated with one or more confidence values ​​from confidence map 235, the one or more confidence values ​​corresponding to one or more pixels adjacent to the first pixel. The confidence level may be referred to as the confidence level or degree of confidence.

[0159] At operation 1425, the image magnification technique includes: magnifying a second pixel within a second image region based on a second filter size and one or more weights associated with one or more confidence values ​​from a confidence map, the one or more confidence values ​​corresponding to one or more pixels adjacent to the second pixel.

[0160] The image magnification technique shown in flowchart 1400 may also include any operation shown or discussed in any of the figures in flowcharts 1200, 1250 and / or 1300.

[0161] In some cases, one or more operations 1405-1435 of flowchart 1400 can be performed by the device that performs one or more operations 1205-1235 of flowchart 1200. In some cases, Figure 14 Image magnification techniques can be Figure 12A It is part of the image processing technology. Figure 12AThe image processing techniques can represent at least some of the operations of the classification engine 220 and / or ISP 240. Figure 14 The image magnification technique can represent at least some of the operations in the operation of the category diagram amplifier (CMUS) 905 that can be used in amplifier 310.

[0162] In some cases, at least a subset of the techniques illustrated by flowcharts 1200, 1250, 1300, and 1400 can be remotely executed by one or more web servers of a cloud service. In some examples, the processes described herein (e.g., processes including operations 200, 300, 400, 700, 900, 1100, 1200, 1250, 1300, 1400, and / or other processes described herein) can be executed by a computing device or apparatus. In one example, processes 200, 300, 400, 700, 900, 1100, 1200, 1250, 1300, and / or 1400 can be executed by... Figure 1 The image capture device 105A performs this operation. In another example, the process including operations 200, 300, 400, 700, 900, 1100, 1200, 1250, 1300 and / or 1400 can be performed by... Figure 1 The image processing device 105B performs the operation. Processes including operations 200, 300, 400, 700, 900, 1100, 1200, 1250, 1300 and / or 1400 can also be performed by... Figure 1 The image capture and processing system 100 performs the operation. The processes including operations 200, 300, 400, 700, 900, 1100, 1200, 1250, 1300 and / or 1400 can be performed by a system having... Figure 15The computing device is configured to perform the architecture of the computing system 1500 shown herein. The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, a computing device for an autonomous vehicle or autonomous vehicle, a robotic device, a television set, and / or any other computing device having the resource capability to perform the processes described herein (including processes involving operations 200, 300, 400, 700, 900, 1100, 1200, 1250, 1300, and / or 1400). In some cases, the computing device or apparatus may include various components such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface can be configured to transmit and / or receive Internet Protocol (IP) based data or other types of data.

[0163] Components of a computing device can be implemented using circuitry. For example, components may include and / or may be implemented using circuitry or other electronic hardware, which may include one or more programmable circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable circuitry) and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0164] The processes illustrated in concept diagrams and flowcharts 200, 300, 400, 700, 900, 1100, 1200, 1250, 1300, and 1400 are organized into logical flowcharts. The operations represented in these logical flowcharts are a series of operations that can be implemented using hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media, which, when executed by one or more processors, perform the described operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement these processes.

[0165] Furthermore, the processes illustrated in conceptual diagrams and flowcharts 200, 300, 400, 700, 900, 1100, 1200, 1250, 1300, and 1400, and / or other processes described herein, can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, implemented in hardware, or a combination thereof. As mentioned above, the code can be stored, for example, in the form of a computer program comprising multiple instructions executable by one or more processors, on a computer-readable or machine-readable storage medium. The computer-readable or machine-readable storage medium can be non-transitory.

[0166] Figure 15 This is a diagram illustrating an example of a system used to implement certain aspects of the techniques described herein. Specifically, Figure 15 An example of a computing system 1500 is shown. The computing system 1500 can be any computing device, such as constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1505. Connection 1505 can be a physical connection using a bus, or a direct connection to processor 1510 (such as in a chipset architecture). Connection 1505 can also be a virtual connection, a network connection, or a logical connection.

[0167] In some embodiments, the computing system 1500 is a distributed system, wherein the functions described herein may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a plurality of such components, each performing some or all of the functions described for that component. In some embodiments, these components may be physical or virtual devices.

[0168] Example system 1500 includes at least one processing unit (CPU or processor) 1510 and connection 1505, which couples various system components, including system memory 1515 such as read-only memory (ROM) 1520 and random access memory (RAM) 1525, to processor 1510. Computing system 1500 may include a cache 1512 of high-speed memory, which is directly connected to, close to, or integrated into processor 1510.

[0169] Processor 1510 may include any general-purpose processor and hardware or software services configured to control processor 1510 (such as services 1532, 1534, and 1536 stored in storage device 1530), as well as dedicated processors in which software instructions are incorporated into the actual processor design. Processor 1510 may essentially be a fully self-contained computing system, containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0170] To enable user interaction, the computing system 1500 includes an input device 1545 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 1500 may also include an output device 1535, which can be one or more of multiple output mechanisms. In some cases, a multi-mode system allows the user to provide multiple types of input / output to communicate with the computing system 1500. The computing system 1500 may include a communication interface 1540, which typically controls and manages user input and system output. The communication interface may use wired and / or wireless transceivers to perform or facilitate the reception and / or transmission of wired or wireless communications, including those utilizing the following: audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple® Lightning® ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, BLUETOOTH® wireless signal transmission, BLUETOOTH® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communication (DSRC) wireless signal transmission, and 802.11. The communication interface 1540 may include Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), microwave access global interoperability (WiMAX), infrared (IR) wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1540 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing system 1500 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S. Global Positioning System (GPS), the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS. There are no restrictions on operation for any particular hardware layout, and therefore the basic functionality here can be easily replaced with an improved hardware or firmware layout when it is developed.

[0171] Storage device 1530 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as magnetic tape, flash memory cards, solid-state storage devices, digital multifunction disks, magnetic tape, floppy disks, hard disks, magnetic tape, strips, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM, rewritable CD, DVD, Blu-ray Disc, holographic disc, another optical medium, secure digital (SD) cards, microSD cards, Memory Stick® cards, smart... Card chips, EMV chips, SIM cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette, and / or combinations thereof.

[0172] Storage device 1530 may include software services, servers, services, etc., which cause the system to perform functions when processor 1510 executes code defining such software. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium, which are connected to necessary hardware components (such as processor 1510, connection 1505, output device 1535, etc.) for performing that function.

[0173] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and excludes carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as compact optical discs (CDs) or digital versatile optical discs (DVDs), flash memory, memory, or memory devices. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or sent using any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0174] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0175] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that these embodiments can be practiced without these specific details. For clarity, in some instances, the techniques described herein may be presented as comprising individual functional blocks including devices, device components, steps or routines in a software-embodied method, or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure these embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring these embodiments.

[0176] The various embodiments described above may be presented as processes or methods, depicted as flowcharts, schematic diagrams, data flow diagrams, structural diagrams, or block diagrams. While a flowchart may describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. A process terminates upon completion of its operations, but may have additional steps not included in the diagram. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0177] The processes and methods described in the examples above can be implemented using computer-executable instructions, which are stored in or otherwise made available from a computer-readable medium. Such instructions may include, for example, instructions or data that cause a general-purpose computer, special-purpose computer, or processing device to perform or otherwise configure it to perform a particular function or a particular set of functions. The portion of the computer resources used may be accessible via a network. Computer-executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include hard disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0178] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may employ any of a variety of form factors. When implemented using software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored on a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By further example, such functionality may also be implemented on a circuit board between different chips or different processes executed in a single device.

[0179] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example modules for providing the functionality described in this disclosure.

[0180] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative embodiments of this application have been described in detail herein, it should be understood that the inventive concept may be embodied and employed in other ways, and the appended claims are intended to be construed as including such variations, in addition to those limited by the prior art. Various features and aspects of the above-described applications may be used individually or collectively. Furthermore, embodiments may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. For illustrative purposes, the methods have been described in a particular order. It should be understood that, in alternative embodiments, the methods may be performed in a different order than that described.

[0181] Those skilled in the art will understand that, without departing from the scope of this specification, the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("<") respectively. ") and greater than or equal to (" Replace with the ") symbol.

[0182] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing a circuit or other hardware to perform the operation, programming a programmable circuit (e.g., a microprocessor or other suitable circuit) to perform the operation, or any combination thereof.

[0183] The phrase “coupled to” refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0184] The language of a claim that states "at least one" and / or "one or more" in a set, or other languages, indicates that one or more members of that set (in any combination) satisfy the claim. For example, the language of a claim stating "at least one of A and B" means A, B, or A and B. In another example, the language of a claim stating "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The use of "at least one" and / or "one or more" in a language set does not limit the set to items listed in that set. For example, the language of a claim stating "at least one of A and B" could mean A, B, or A and B, and could additionally include items not listed in the set of A and B.

[0185] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been generally described above regarding their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0186] The techniques described herein can also be implemented using electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, mobile phones with wireless communication devices, or integrated circuit devices with multiple uses (including applications in mobile phones with wireless communication devices and other devices). Any feature described as a module or component can be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques can be implemented at least in part by a computer-readable data storage medium comprising program code that, when executed, performs one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Alternatively or concurrently, the technology may be implemented, at least in part, by a computer-readable communication medium (such as a propagating signal or wave) that carries or transmits program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0187] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0188] The illustrative aspects of this disclosure include:

[0189] Aspect 1: A method for processing video data. The method includes: receiving image data captured by an image sensor; determining a first image region in the image data that depicts a first object category among a plurality of object categories; determining a second image region in the image data that depicts a second object category among the plurality of object categories; and generating an image based on the image data using the image capture process by applying a first setting for the image capture process to the first image region and by applying a second setting for the image capture process to the second image region.

[0190] Aspect 2: The method according to aspect 1 further includes: generating a category map by dividing the image data into a plurality of image regions including the first image region and the second image region, wherein each of the plurality of image regions corresponds to one of the plurality of object categories; identifying the first object category corresponding to the first setting for the image capture process; and identifying the second object category corresponding to the second setting for the image capture process.

[0191] Aspect 3: The method according to any one of Aspect 1 or 2 further includes: generating a scaled-down copy of the image data by scaling down the image data, wherein generating the category map based on the image data includes generating the category map based on the scaled-down copy of the image data.

[0192] Aspect 4: The method according to any one of Aspects 1 to 3 further comprises: generating a confidence map based on the image data, the confidence map identifying multiple confidence levels corresponding to multiple portions of the image data, wherein each of the multiple confidence levels identifies a confidence level describing one of the multiple object categories with respect to a corresponding portion of the multiple portions, and wherein generating the image includes processing the image data using an image signal processor (ISP) tuned based on the category map and the confidence map.

[0193] Aspect 5: The method according to any one of aspects 1 to 4 further includes: enlarging the category map.

[0194] Aspect 6: The method according to any one of Aspects 1 to 5, wherein enlarging the category map comprises: enlarging the category map to a size that matches the size of at least one of the image data and the image.

[0195] Aspect 7: The method according to any one of Aspects 1 to 6, wherein magnifying the category map is performed using nearest neighbor magnification.

[0196] Aspect 8: The method according to any one of Aspects 1 to 7, wherein magnifying the category map is performed using nearest neighbor magnification modified with spatial weight filtering.

[0197] Aspect 9: The method according to any one of Aspects 1 to 8, wherein amplifying the category map using the nearest neighbor amplification modified by spatial weighted filtering comprises: identifying a first filter size corresponding to the first image region and a second filter size corresponding to the second image region, wherein the first filter size is smaller than the second filter size; amplifying the first pixel based on the first filter size and one or more weights associated with one or more confidence values ​​from the confidence map corresponding to one or more pixels adjacent to the first pixel in the first image region; and amplifying the second pixel based on the second filter size and one or more weights associated with one or more confidence values ​​from the confidence map corresponding to one or more pixels adjacent to the second pixel in the second image region.

[0198] Aspect 10: The method according to any one of Aspects 1 to 9, wherein the image capture process includes processing the image data using an image signal processor (ISP), wherein the first setting for the image capture process is a first tuning setting for the ISP, and wherein the second setting for the image capture process is a second tuning setting for the ISP.

[0199] Aspect 11: The method according to any one of Aspects 1 to 10, wherein the first tuning setting and the second tuning setting include applying different intensities of the noise reduction ISP tuning parameters during the processing of the image data.

[0200] Aspect 12: The method according to any one of Aspects 1 to 11, wherein the first tuning setting and the second tuning setting include applying different intensities of sharpening ISP tuning parameters during the processing of the image data.

[0201] Aspect 13: The method according to any one of Aspects 1 to 12, wherein the first tuning setting and the second tuning setting include applying different intensities of the color saturation ISP tuning parameters during the processing of the image data.

[0202] Aspect 14: The method according to any one of Aspects 1 to 13, wherein the first tuning setting and the second tuning setting include applying different intensities of tone mapping ISP tuning parameters during the processing of the image data.

[0203] Aspect 15: The method according to any one of Aspects 1 to 14, wherein the first tuning setting and the second tuning setting include applying different intensities of the gamma ISP tuning parameters during the processing of the image data.

[0204] Aspect 16: The method according to any one of Aspects 1 to 15, wherein the ISP tuning settings for the image signal processor (ISP) are used to set at least one value associated with at least one of the following: the denoising module of the ISP, the sharpening module of the ISP, the tone mapping module of the ISP, the color saturation module of the ISP, the gamma module of the ISP, the blurring module of the ISP, the de-mosaic module of the ISP, the color space conversion module of the ISP, the gain module of the ISP, the brightness module of the ISP, the shading module of the ISP, the edge enhancement module of the ISP, the image compositing module for high dynamic range (HDR) of the ISP, the special effects processing module of the ISP, the artificial noise (e.g., grain) adder module of the ISP, the edge-oriented magnification module of the ISP, the autofocus module of the ISP, the auto exposure module of the ISP, the auto white balance module of the ISP, the aperture control module of the ISP, the shutter speed control module of the ISP, the ISO control module of the ISP, the lens position module of the ISP, the electronic image stabilization module of the ISP, and the flash control module of the ISP.

[0205] Aspect 17: The method according to any one of aspects 1 to 16 further includes: generating one or more modification amounts, the one or more modification amounts identifying at least one of the following: a first deviation from the default tuning setting applied by the ISP in the first image region during the processing of the image data, and a second deviation from the default tuning setting applied by the ISP in the second image region during the processing of the image data.

[0206] Aspect 18: The method according to any one of Aspects 1 to 17, wherein the one or more modification amounts identify at least one of the first deviation and the second deviation by multiplying the one or more modification amounts by the default ISP tuning setting.

[0207] Aspect 19: The method according to any one of Aspects 1 to 18, wherein the one or more modification amounts identify at least one of the first deviation and the second deviation by adding the one or more modification amounts to the default ISP tuning setting.

[0208] Aspect 20: The method according to any one of aspects 1 to 19, wherein the one or more modification amounts identify at least one of the first deviation and the second deviation based on an increment in a predetermined list of possible ISP tuning settings including the default ISP tuning setting, the increment being based on the modification amount.

[0209] Aspect 21: The method according to any one of aspects 1 to 20 further includes: generating a category map by dividing the image data into a plurality of image regions including the first image region and the second image region, wherein each of the plurality of image regions corresponds to one of the plurality of object categories, wherein the one or more modifications are generated at least based on the category map.

[0210] Aspect 22: The method according to any one of aspects 1 to 21 further includes: reducing the category map before generating the one or more modifications based on the category map.

[0211] Aspect 23: The method according to any one of aspects 1 to 22 further comprises: generating one or more mixed modification amounts by mixing the one or more modification amounts with information corresponding to a confidence map, the confidence map identifying multiple confidence levels corresponding to multiple portions of the image data, wherein each of the multiple confidence levels identifies the degree of confidence in determining that a corresponding portion of the multiple portions depicts one of the multiple object categories.

[0212] Aspect 24: The method according to any one of aspects 1 to 23 further includes: generating one or more filtered modification amounts by filtering the one or more mixed modification amounts using a low-pass filter.

[0213] Aspect 25: The method according to any one of aspects 1 to 24 further includes: generating one or more amplified modification amounts by amplifying the one or more filtered modification amounts.

[0214] Aspect 26: The method according to any one of aspects 1 to 25, wherein processing the image data using the ISP comprises: processing the image data using at least one of the following: the one or more modification amounts, the one or more mixed modification amounts, the one or more filtered modification amounts, and the one or more amplified modification amounts.

[0215] Aspect 27: The method according to any one of Aspects 1 to 26, wherein at least one of the first setting for the image capture process and the second setting for the image capture process is a tuning position associated with at least one of the following: lens position, flash, focus, exposure, white balance, aperture size, shutter speed, ISO, analog gain, digital gain, noise reduction, sharpening, tone mapping, color saturation, demosaic, color space conversion, colorization, edge enhancement, image combination for high dynamic range (HDR), effects, grain addition, artificial noise addition, edge-oriented magnification, magnification, reduction, and electronic image stabilization.

[0216] Aspect 28: The method according to any one of aspects 1 to 27, wherein the image data is raw image data.

[0217] Aspect 29: The method according to any one of aspects 1 to 28 further includes: de-mosaicing the image data.

[0218] Aspect 30: The method according to any one of aspects 1 to 29 further includes: converting the image data from a first color space to a second color space.

[0219] Aspect 31: The method according to any one of aspects 1 to 30, wherein the second color space is the YUV color space.

[0220] Aspect 32: An apparatus for image processing, the apparatus comprising: one or more memory units storing instructions; and one or more processors executing the instructions, wherein execution of the instructions by the one or more processors causes the one or more processors to: perform the method according to any one of aspects 1 to 31.

[0221] Aspect 33: The apparatus according to aspect 32, wherein the apparatus is a mobile device.

[0222] Aspect 34: The apparatus according to any one of aspects 32 or 33, wherein the apparatus is a wireless communication device.

[0223] Aspect 35: The apparatus according to any one of aspects 32 to 34, wherein the apparatus is a camera comprising at least an image sensor and the one or more processors.

[0224] Aspect 36: The apparatus according to any one of aspects 32 to 35, wherein the one or more processors include an image signal processor (ISP).

[0225] Aspect 37: The apparatus according to any one of aspects 32 to 36, wherein the one or more processors include a classification engine.

[0226] Aspect 38: The apparatus according to any one of aspects 32 to 37, wherein the apparatus includes a display configured to display the image.

[0227] Aspect 39: A non-transitory computer-readable storage medium having a program embodied thereon, wherein the program is executable by a processor to perform a method of image processing, the method comprising: the method according to any one of aspects 1 to 31.

[0228] Aspect 40: An apparatus for image processing, the apparatus comprising: a unit for performing the method according to any one of aspects 1 to 31.

[0229] Aspect 41: An apparatus for image processing, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: receive image data captured by an image sensor; determine that a first object image region in the image data depicts a first object category among a plurality of object categories; determine that a second object image region in the image data depicts a second object category among the plurality of object categories; identify a plurality of confidence levels corresponding to a plurality of confidence image regions in the image data, wherein each of the plurality of confidence levels identifies a confidence level regarding the depiction of an object category among the plurality of object categories by the corresponding confidence image region in the plurality of confidence image regions; and generate an image based on the image data using an image capture process, including by applying different settings for the image capture process to different portions of the image data, the different portions of the image data being identified based on the first object image region, the second object image region, and the plurality of confidence image regions.

[0230] Aspect 42: The apparatus according to aspect 41, wherein the one or more processors are configured to: generate one or more modification amounts, the one or more modification amounts identifying at least one of the following: a first deviation for the first object image region from the default setting for the image capture process, and a second deviation for the second object image region from the default setting for the image capture process, wherein the different settings for the image capture process are based on the one or more modification amounts.

[0231] Aspect 43: The apparatus according to any one of aspects 41 to 42, wherein the one or more processors are configured to: adjust the one or more modification amounts, including mixing the one or more modification amounts with a mixed update based on a plurality of confidence levels corresponding to the plurality of confidence image regions, wherein mixing the one or more modification amounts with the mixed update is used to adjust at least one of the first deviation and the second deviation in at least one region of the image data.

[0232] Aspect 44: An apparatus according to any one of aspects 41 to 43, wherein the one or more processors are configured to: generate a category map that divides the image data into a plurality of object image regions including a first object image region and a second object image region, wherein each of the plurality of object image regions corresponds to an object category in the plurality of object categories; identify the first object category corresponding to a first setting for the image capture process; and identify the second object category corresponding to a second setting for the image capture process.

[0233] Aspect 45: An apparatus according to any one of aspects 41 to 44, wherein the one or more processors are configured to: generate a confidence map that divides the image data into a plurality of confidence image regions corresponding to the plurality of confidence levels, the different portions of the image data being identified based on the category map and the confidence map.

[0234] Aspect 46: The apparatus according to any one of aspects 41 to 45, wherein the image capture process includes processing the image data using an image signal processor (ISP) of the one or more processors, wherein the different settings for the image capture process are different tuning settings for the ISP.

[0235] Aspect 47: The apparatus according to any one of aspects 41 to 46, wherein the different tuning settings for the ISP include applying different intensities of ISP tuning parameters during processing of the image data using the ISP, wherein the ISP tuning parameters are one of noise reduction, sharpening, color saturation, color mapping, color processing, and tone mapping.

[0236] Aspect 48: The method according to any one of Aspects 41 to 47, wherein the different tuning settings for the ISP include applying different intensities of ISP tuning parameters during processing of the image data using the ISP, wherein the ISP tuning parameters are one of noise reduction, sharpening, color saturation, color mapping, color processing, and tone mapping.

[0237] Aspect 49: The apparatus according to any one of aspects 41 to 48, wherein the different settings include settings associated with at least one of the following: lens position, flash, focus, exposure, white balance, aperture size, shutter speed, ISO, analog gain, digital gain, noise reduction, sharpening, tone mapping, color saturation, demosaic, color space conversion, colorization, edge enhancement, image combination for high dynamic range (HDR), effects, artificial noise addition, edge-oriented magnification, magnification, reduction, and electronic image stabilization.

[0238] Aspect 50: An apparatus according to any one of aspects 41 to 49, wherein the one or more processors are configured to process the image data, including at least one of the following operations: depixelating the image data and converting the image data from a first color space to a second color space.

[0239] Aspect 51: An apparatus according to any one of aspects 41 to 50, wherein the one or more processors are configured to: receive user input associated with at least one of the first object image region and the second object image region, wherein at least one of the different settings is defined based on the user input and corresponds to one of the first object image region and the second object image region.

[0240] Aspect 52: The apparatus according to any one of aspects 41 to 51, wherein the one or more processors include an image signal processor (ISP) that applies the different settings for the image capture process to the different portions of the image data.

[0241] Aspect 53: The apparatus according to any one of aspects 41 to 52, wherein the one or more processors include a classification engine for identifying at least the first object image region and the second object image region, wherein the classification engine is at least partially located on an integrated circuit chip.

[0242] Aspect 54: The apparatus according to any one of aspects 41 to 53, wherein the apparatus is one of a mobile device, a wireless communication device, and a camera.

[0243] Aspect 55: The apparatus according to any one of aspects 41 to 54 further includes: an image sensor.

[0244] Aspect 56: The apparatus according to any one of aspects 41 to 55 further includes: a display for displaying the image.

[0245] Aspect 57: An image processing method, the method comprising: receiving image data captured by an image sensor; determining that a first object image region in the image data depicts a first object category among a plurality of object categories; determining that a second object image region in the image data depicts a second object category among the plurality of object categories; identifying a plurality of confidence levels corresponding to a plurality of confidence image regions in the image data, wherein each of the plurality of confidence levels identifies a confidence level regarding the depiction of an object category among the plurality of object categories by the corresponding confidence image region in the plurality of confidence image regions; and generating an image based on the image data using an image capture process, including by applying different settings for the image capture process to different portions of the image data, the different portions of the image data being identified based on the first object image region, the second object image region, and the plurality of confidence image regions.

[0246] Aspect 58: The method according to aspect 57 further includes: generating one or more modification amounts, the one or more modification amounts identifying at least one of the following: a first deviation for the first object image region from the default settings for the image capture process, and a second deviation for the second object image region from the default settings for the image capture process, wherein the different settings for the image capture process are based on the one or more modification amounts.

[0247] Aspect 59: The method according to any one of Aspects 57 to 58, further comprising: adjusting the one or more modification amounts, including mixing the one or more modification amounts with a mixed update based on the plurality of confidence levels corresponding to the plurality of confidence image regions, wherein mixing the one or more modification amounts with the mixed update is used to adjust at least one of the first deviation and the second deviation in at least one region of the image data.

[0248] Aspect 60: The method according to any one of aspects 57 to 59 further includes: generating a category map that divides the image data into a plurality of object image regions including a first object image region and a second object image region, wherein each of the plurality of object image regions corresponds to an object category among the plurality of object categories; identifying the first object category corresponding to a first setting for the image capture process; and identifying the second object category corresponding to a second setting for the image capture process.

[0249] Aspect 61: The method according to any one of aspects 57 to 60 further includes: generating a confidence map, the confidence map dividing the image data into a plurality of confidence image regions corresponding to the plurality of confidence levels, the different portions of the image data being identified based on the category map and the confidence map.

[0250] Aspect 62: The method according to any one of aspects 57 to 61, wherein the image capture process includes processing the image data using an image signal processor (ISP) of the one or more processors, wherein the different settings for the image capture process are different tuning settings for the ISP.

[0251] Aspect 63: The method according to any one of Aspects 57 to 62, wherein the different settings include settings associated with at least one of the following: lens position, flash, focus, exposure, white balance, aperture size, shutter speed, ISO, analog gain, digital gain, noise reduction, sharpening, tone mapping, color saturation, demosaic, color space conversion, colorization, edge enhancement, image combination for high dynamic range (HDR), effects, grain addition, artificial noise addition, edge-oriented magnification, magnification, reduction, and electronic image stabilization.

[0252] Aspect 64: The method according to any one of aspects 57 to 63 further includes: processing the image data, including at least one of the following operations: depixelating the image data and converting the image data from a first color space to a second color space.

[0253] Aspect 65: The method according to any one of aspects 57 to 64 further includes: receiving user input associated with at least one of the first object image region and the second object image region, wherein at least one of the different settings is defined based on the user input and corresponds to one of the first object image region and the second object image region.

[0254] Aspect 66: The method according to any one of aspects 57 to 65, wherein applying the different settings for the image capture process to the different portions of the image data comprises: using an image signal processor (ISP) to apply the different settings for the image capture process to the different portions of the image data.

[0255] Aspect 67: The method according to any one of Aspects 57 to 66, wherein identifying the first object image region and the second object image region comprises: using a classification engine to identify the first object image region and the second object image region, the classification engine being at least partially located on an integrated circuit chip.

[0256] Aspect 68: The method according to any one of aspects 57 to 67 further includes: displaying the image on a display.

[0257] Aspect 69: A non-transitory computer-readable storage medium having a program embodied thereon, the program being executable by a processor to perform a method of image processing, the method comprising: receiving image data captured by an image sensor; determining that a first object image region in the image data depicts a first object category among a plurality of object categories; determining that a second object image region in the image data depicts a second object category among the plurality of object categories; identifying a plurality of confidence levels corresponding to a plurality of confidence image regions in the image data, wherein each of the plurality of confidence levels identifies a confidence level regarding the depiction of an object category among the plurality of object categories by the corresponding confidence image region in the plurality of confidence image regions; and generating an image based on the image data using an image capture process, including by applying different settings for the image capture process to different portions of the image data, the different portions of the image data being identified based on the first object image region, the second object image region, and the plurality of confidence image regions.

Claims

1. An apparatus for image processing, the apparatus comprising: At least one memory; as well as At least one processor coupled to the at least one memory, the at least one processor being configured to: The category regions of an image are classified to identify the categories of objects depicted in those categories. The confidence region of the image is associated with at least a confidence level that classifies the category region to the object category, wherein the category region and the confidence region intersect at an intersection region of the image; as well as The intersecting regions of the image are processed using image processing settings to generate a processed image.

2. The apparatus according to claim 1, wherein, The at least one processor is configured to: Generate a modification amount associated with the intersecting region of the image, wherein the modification amount identifies a deviation from the default image processing settings, wherein the image processing settings are based on applying the deviation to the default image processing settings.

3. The apparatus according to claim 2, wherein, The default image processing settings are the default settings associated with the image.

4. The apparatus according to claim 2, wherein, The default image processing settings are default settings associated with the image capture device, wherein the image is captured using the image capture device.

5. The apparatus according to claim 2, wherein, The default image processing setting identifies the default intensity at which a specific image processing function is to be applied, and wherein the deviation from the default image processing setting includes the deviation from the default intensity at which the specific image processing function is to be applied.

6. The apparatus according to claim 2, wherein, The modification includes an offset from the default image processing settings.

7. The apparatus according to claim 2, wherein, The modification amount includes the multiplier of the default image processing settings.

8. The apparatus according to claim 1, wherein, The at least one processor is configured to: The image is classified into multiple category regions to identify multiple object categories depicted across the multiple category regions of the image, wherein the multiple category regions include the category regions; The image is associated with multiple confidence regions and multiple confidence levels that classify the multiple category regions into multiple object categories, wherein the category regions and the confidence regions intersect at an intersection region of the image, and wherein the multiple confidence regions include the confidence regions; and The intersecting regions of the image are processed using image processing settings to generate a processed image.

9. The apparatus according to claim 1, wherein, The at least one processor is configured to: Generate a classification map that maps multiple object categories to multiple category regions of the image, wherein the multiple category regions include the category regions; Generate a confidence map that maps multiple confidence levels to multiple confidence regions of the image, wherein the multiple confidence regions include the confidence regions; and The category map is combined with the confidence map to generate a combined map that maps information indicating multiple image processing settings to multiple intersecting regions of the image, wherein the multiple image processing settings include the image processing settings, wherein the multiple intersecting regions include the intersecting regions, and wherein, in order to process the intersecting regions of the image using the image processing settings, the at least one processor is configured to process the multiple intersecting regions of the image using the respective image processing settings of the multiple image processing settings.

10. The apparatus according to claim 9, wherein, The information indicating the plurality of image processing settings includes a plurality of modification amounts associated with the plurality of intersecting regions of the image, wherein the plurality of modification amounts identify a plurality of deviations from the default image processing settings, wherein the plurality of image processing settings are based on applying the plurality of deviations to the default image processing settings.

11. The apparatus according to claim 9, wherein, The at least one processor is configured to: The combined image is filtered using at least one of the following: low-pass filter, Gaussian filter, average filter, box blur filter, lens blur filter, radial blur filter, motion blur filter, shape blur filter, smart blur filter, surface blur filter, blur filter, scaling filter, or resampling filter.

12. The apparatus according to claim 9, wherein, The at least one processor is configured to: The combined graph is magnified using an amplification algorithm modified by spatial weight filtering.

13. The apparatus according to claim 1, wherein, The image includes raw image data, and wherein, in order to process the intersecting regions of the image using the image processing settings, the at least one processor is configured to use an image signal processor (ISP) to process the raw image data using the image processing settings.

14. The apparatus according to claim 13, wherein, The image processing settings are associated with at least one of the following: noise reduction, sharpening, color saturation, color mapping, color processing, or tone mapping.

15. The apparatus according to claim 13, wherein, The image processing settings are associated with at least one of the following: lens position, flash, focus, exposure, white balance, aperture size, shutter speed, ISO, analog gain, digital gain, noise reduction, sharpening, tone mapping, color saturation, demosaic, color space conversion, colorization, edge enhancement, image combination for high dynamic range (HDR), effects, artificial noise addition, edge-oriented magnification, magnification, reduction, and electronic image stabilization.

16. The apparatus according to claim 1, wherein, The device is one of a mobile device, a wireless communication device, and a camera.

17. The apparatus of claim 15, further comprising: A display configured to display the processed image.

18. A method for image processing, the method comprising: The category regions of an image are classified to identify the categories of objects depicted in those categories. The confidence region of the image is associated with at least a confidence level that classifies the category region to the object category, wherein the category region and the confidence region intersect at an intersection region of the image; as well as The intersecting regions of the image are processed using image processing settings to generate a processed image.

19. The method of claim 18, further comprising: Generate a classification map that maps multiple object categories to multiple category regions of the image, wherein the multiple category regions include the category regions; Generate a confidence map that maps multiple confidence levels to multiple confidence regions of the image, wherein the multiple confidence regions include the confidence regions; and The category map is combined with the confidence map to generate a combined map that maps information indicating multiple image processing settings to multiple intersecting regions of the image, wherein the multiple image processing settings include the image processing settings, wherein the multiple intersecting regions include the intersecting regions, and wherein, in order to process the intersecting regions of the image using the image processing settings, the respective image processing settings of the multiple image processing settings are used to process the multiple intersecting regions of the image.

20. The method according to claim 18, wherein, The image processing settings are associated with at least one of the following: noise reduction, sharpening, color saturation, color mapping, color processing, tone mapping, lens position, flash, focus, exposure, white balance, aperture size, shutter speed, ISO, analog gain, digital gain, demosaic, color space conversion, colorization, edge enhancement, high dynamic range (HDR) image combination, effects, artificial noise addition, edge-oriented magnification, magnification, reduction, and electronic image stabilization.

21. An apparatus for image processing, the apparatus comprising: At least one memory; as well as At least one processor coupled to the at least one memory, the at least one processor being configured to: Receive image data captured by the image sensor; Determine a first image region in the image data to depict a first object category among multiple object categories; Determine a second image region in the image data to depict a second object category among the plurality of object categories; as well as An image is generated based on the image data using the image capture process by applying a first setting for the image capture process to the first image region and by applying a second setting for the image capture process to the second image region.

22. A method for processing video data, the method comprising: Receive image data captured by the image sensor; Determine a first image region in the image data to depict a first object category among multiple object categories; Determine a second image region in the image data to depict a second object category among the plurality of object categories; as well as An image is generated based on the image data using the image capture process by applying a first setting for the image capture process to the first image region and by applying a second setting for the image capture process to the second image region.

23. A non-transitory computer-readable storage medium having a program embodied thereon, the program being executable by a processor to perform a method of image processing, the method comprising: Receive image data captured by the image sensor; Determine a first image region in the image data to depict a first object category among multiple object categories; Determine a second image region in the image data to depict a second object category among the plurality of object categories; as well as An image is generated based on the image data using the image capture process by applying a first setting for the image capture process to the first image region and by applying a second setting for the image capture process to the second image region.

Images