Automatic Camera Debugging

Abstract

Techniques and systems are provided for determining one or more camera settings. For example, an indication of an image quality metric selected for adjustment can be received, and a target image quality metric value for the selected image quality metric can be determined. A data point can be determined from among a plurality of data points. The data point corresponds to a camera setting having an image quality metric value closest to the target image quality metric value.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present application claims the priority of Indian Application No. 202041013885, filed provisionally in India on March 30, 2020, titled “AUTOMATED CAMERATUNING”, which is incorporated herein by reference in its entirety and for all purposes. Technical Field

[0003] The present disclosure relates generally to camera commissioning, and more particularly to techniques and systems for performing automatic camera commissioning based on user feedback. Background Art

[0004] An image capture device, such as a camera, can use an image sensor to receive light and capture image frames, such as still images or video frames. The image capture device may include a processor (e.g., one or more image signal processors (ISPs)) that can receive and process one or more image frames. For example, a raw image frame captured by an image sensor can be processed by an ISP to generate a final image.

[0005] The ISP may process the captured image frames by applying a plurality of modules to the captured image frames. Each module may include a large number of debuggable parameters (such as hundreds or thousands of parameters for each module). In addition, the modules may be interdependent in the case where different modules may affect similar aspects of the image. For example, both denoising and texture correction or enhancement may affect high frequency aspects of the image. Therefore, a large number of parameters are determined or adjusted for the ISP to generate a final image from the captured raw image. Summary of the invention

[0006] Systems and techniques for performing automatic camera tuning for determining one or more camera settings based on user feedback are described herein. According to an illustrative example, a method for determining one or more camera settings is provided. The method includes: receiving an indication of selecting an image quality metric for adjustment; determining a target image quality metric value for the selected image quality metric; and determining, from a plurality of data points, a data point corresponding to a camera setting having an image quality metric value closest to the target image quality metric value.

[0007] In another example, an apparatus for determining one or more camera settings is provided. The apparatus includes a memory configured to store at least one image and one or more processors implemented in circuitry and coupled to the memory. The one or more processors are configured and may: receive an indication of selecting an image quality metric for adjustment; determine a target image quality metric value for the selected image quality metric; and determine, from a plurality of data points, a data point corresponding to a camera setting having an image quality metric value closest to the target image quality metric value.

[0008] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: receive an indication of selecting an image quality metric for adjustment; determine a target image quality metric value for the selected image quality metric; and determine, from a plurality of data points, a data point corresponding to a camera setting having an image quality metric value that is closest to the target image quality metric value.

[0009] In another example, an apparatus for determining one or more camera settings is provided. The apparatus includes: means for receiving an indication of selecting an image quality metric for adjustment; means for determining a target image quality metric value for the selected image quality metric; and means for determining, from a plurality of data points, a data point corresponding to a camera setting having an image quality metric value closest to the target image quality metric value.

[0010] In some aspects, the indication of selecting the image quality metric includes a direction of adjustment. In some aspects, the direction of adjustment includes a decrease in the image quality metric or an increase in the image quality metric.

[0011] In some aspects, the above methods, apparatus, and computer-readable media further include removing one or more data points having the same metric value for the selected image quality metric from the plurality of data points.

[0012] In some aspects, the methods, apparatus, and computer-readable media described above further include receiving an indication of a selection of a particular camera setting for adjustment, wherein the selected image quality metric is associated with the selected particular camera setting.

[0013] In some aspects, the methods, apparatus, and computer-readable media described above further include removing, from the plurality of data points, one or more data points corresponding to one or more camera settings having a lower score than the selected particular camera setting.

[0014] In some aspects, the above methods, apparatus, and computer-readable media further include removing from the plurality of data points one or more data points corresponding to one or more camera settings having the same metric value for the selected image quality metric and having a lower score than the selected specific camera setting.

[0015] In some aspects, the methods, apparatus, and computer-readable media described above further include: determining, based on an indication of selecting an image quality metric, that a direction of adjustment for the image quality metric includes a reduction in the image quality metric; and removing, from a plurality of data points, one or more data points corresponding to one or more camera settings having greater metric values ​​for the selected image quality metric than the selected particular camera setting.

[0016] In some aspects, removing one or more data points from the plurality of data points generates a set of data points. In some aspects, the above methods, apparatus, and computer-readable media further comprise: sorting the set of data points in descending order.

[0017] In some aspects, the above-described methods, apparatus, and computer-readable media further include: determining, based on an indication of selecting an image quality metric, that a direction of adjustment for the image quality metric includes an increase in the image quality metric; and removing, from a plurality of data points, one or more data points corresponding to one or more camera settings having smaller metric values ​​for the selected image quality metric than the selected particular camera setting.

[0018] In some aspects, removing one or more data points from the plurality of data points generates a set of data points. In some aspects, the above methods, apparatus, and computer-readable media further include: sorting the set of data points in ascending order.

[0019] In some aspects, the above-mentioned methods, devices and computer-readable media also include: determining a metric factor based on a metric value of the selected image quality metric, data points among multiple data points having extreme values ​​of the selected image quality metric, and the number of multiple data points; and determining a target image quality metric value for the selected image quality metric based on the metric factor and the metric value of the selected image quality metric.

[0020] In some aspects, the above-mentioned methods, apparatus and computer-readable media also include: receiving an indication of selecting an adjustment strength for an image quality metric; and determining a target image quality metric value for the selected image quality metric based on a metric factor, a metric value of the selected image quality metric, and the adjustment strength for the image quality metric.

[0021] In some aspects, the above-described methods, apparatus, and computer-readable media further include: receiving an indication of selecting a desired number of output camera settings; and determining a target image quality metric value for the selected image quality metric based on a metric factor, a metric value of the selected image quality metric, and the desired number of output camera settings.

[0022] In some aspects, the above-described methods, apparatus, and computer-readable media further include: receiving an indication to select an adjustment strength for an image quality metric; receiving an indication to select a desired number of output camera settings; and determining a target image quality metric value for the selected image quality metric based on a metric factor, a metric value of the selected image quality metric, an adjustment strength for the image quality metric, and a desired number of output camera settings.

[0023] In some aspects, the above methods, apparatus, and computer-readable media further include: outputting information associated with the determined data points for display.

[0024] In some aspects, the methods, apparatus, and computer-readable media described above further include debugging the image signal process using camera settings corresponding to the determined data points.

[0025] In some aspects, selecting the image quality metric for adjustment is based on selecting a graphical element of a graphical user interface. In some aspects, the graphical element includes an option to increase the image quality metric or to decrease the image quality metric. In some aspects, the graphical element is associated with a displayed image having an adjustment value for the image quality metric.

[0026] In some aspects, selecting the image quality metric for adjustment is based on selecting a displayed image frame having an adjustment value for the image quality metric.

[0027] In some aspects, the apparatus includes a camera, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the apparatus includes one or more cameras for capturing one or more image frames. In some aspects, the apparatus also includes a display for displaying one or more image frames, notifications, and / or other displayable data.

[0028] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0029] The foregoing together with other features and embodiments will become more apparent with reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The illustrative embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0031] Figure 1 is a diagram illustrating the architecture of a camera system according to some examples;

[0032] Figure 2 is a diagram illustrating an example of a manual debugging process for debugging image signal processor (ISP) parameters according to some examples;

[0033] Figure 3A and Figure 3B are examples of image frames illustrating expected image quality (IQ) changes resulting from fine-tuning according to some examples;

[0034] Figure 4A and Figure 4B are examples of image frames showing expected IQ changes resulting from fine-tuning according to some examples;

[0035] Figure 5A and Figure 5B is an example of an image frame showing an expected IQ change resulting from fine-tuning according to some examples;

[0036] Figure 6 is a diagram illustrating an example of a graphical user interface of an automatic camera commissioning tool according to some examples;

[0037] Figure 7 is a flow chart illustrating an example of a process for performing automatic camera commissioning according to some examples;

[0038] Figure 8 is a flow chart illustrating an example of a parameter setting search process according to some examples;

[0039] Fig. 9A and Fig. 9B is an image frame showing a comparison between a capture result obtained using coarsely adjusted settings according to some examples and a capture result obtained using finely adjusted settings determined using the techniques described herein;

[0040] Fig. 10A and Fig. 10B is an image frame showing a comparison between a capture result obtained using coarsely adjusted settings according to some examples and a capture result obtained using finely adjusted settings determined using the techniques described herein;

[0041] Fig.11A and Fig. 11B is an image frame showing a comparison between a capture result obtained using coarsely adjusted settings according to some examples and a capture result obtained using finely adjusted settings determined using the techniques described herein;

[0042] Fig.12 is a flow chart illustrating an example of a process for automatic camera commissioning using the techniques described herein, according to some examples;

[0043] Fig.13 is a diagram illustrating an example of a graphical user interface of an automatic camera commissioning tool according to some examples;

[0044] Fig.14A and Fig. 14B is an image frame showing a comparison between capture results obtained using originally tuned settings of a device and capture results obtained using fine-tuned settings of the device determined using the techniques described herein according to some examples;

[0045] Fig.15 is a flowchart illustrating an example of a process for determining one or more camera settings using the techniques described herein, according to some examples; and

[0046] Fig.16 is a block diagram of an example computing device that can be used to implement some aspects of the techniques described herein, according to some examples. DETAILED DESCRIPTION

[0047] Specific aspects and embodiments of the present disclosure are provided below. It will be appreciated by those skilled in the art that some of these aspects and embodiments may be applied independently and some of them may be applied in combination. In the following description, for the purpose of illustration, specific details are set forth to achieve a comprehensive understanding of the embodiments of the present application. However, it is apparent that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be restrictive.

[0048] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments for those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0049] A camera (also referred to as an image capture device) is a device that uses an image sensor (also referred to as a camera sensor) to receive light and capture image frames (such as still images or video frames). A camera may include a processor, such as an image signal processor (ISP), that may receive one or more image frames and process the one or more image frames. For example, a raw image frame captured by an image sensor may be processed by an ISP to generate a final image. The ISP may process the captured image frames by applying a plurality of modules or processing blocks (e.g., filters) to the captured image frames. These modules may include processing blocks for operations including: denoising or noise filtering, edge enhancement (e.g., using a sharpening filter), color balancing, contrast, intensity adjustment (such as dimming or brightening), hue adjustment, lens / sensor noise correction, Bayer filtering (using a Bayer filter), demosaicing, color conversion, correction or enhancement / suppression of image attributes, and the like. Each module may include a large number of adjustable parameters (such as hundreds or thousands of parameters for each module). In addition, modules may be interdependent because different modules may affect similar aspects of an image. For example, both denoising and texture correction or enhancement may affect high-frequency aspects of an image. Thus, a number of parameters are determined or adjusted for the ISP to generate a final image based on the captured raw image.

[0050] The parameters for the ISP are typically manually debugged by an expert who is experienced in how to process input images for the desired output images. Camera debugging can be a time-consuming and resource-intensive process. For example, due to the correlation between the ISP modules (e.g., filters) and the sheer number of adjustable parameters, it may take an expert several weeks (e.g., 3-8 weeks) to determine, test, and / or adjust the device settings for the parameters based on a specific image / camera sensor and ISP combination. Because the camera sensor or other camera features (e.g., lens characteristics or defects, aperture size, shutter speed and movement, flash brightness and color, and / or other features) can affect the captured image and at least some of the adjustable parameters for the ISP. Each image / camera sensor and ISP combination will need to be debugged by an expert.

[0051] Systems, devices, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") that provide automatic camera debugging are described herein. As described in detail herein, the automatic camera debugging systems and techniques can be used to automatically debug an ISP, image / camera sensors, and / or other components of a camera system. In some examples, an automatic camera debugging tool can be used to implement or perform the automatic camera debugging techniques described herein. By interacting with a graphical user interactive (GUI) of the automatic camera debugging tool, the automatic camera debugging tool can be used to perform fine-tuning of the ISP. Further details about the systems and techniques are provided herein with reference to various figures.

[0052] Figure 1 is a diagram showing the architecture of a camera system 100 including a device 101 . Figure 1 The device 101 includes various components, including: a camera controller 125 with an image signal processor (ISP) 120, a processor 135 with a digital signal processor (DSP) 130, a memory 140 for storing instructions 145, a display 150, and an input / output (I / O) component 155. The device 101 can be connected to a power source 160.

[0053] Camera system 100 also includes camera 105. Camera controller 125 may receive image data from camera 105. In some cases, image sensor 115 (also referred to as a camera sensor) of camera 105 may send image data to camera controller 125. Figure 1 As shown, camera 105 includes lens 110. Lens 110 can receive light from a scene including an object. Lens 110 directs light to image sensor 115, which includes a pixel array for generating image frames (also referred to as images or frames). In response to image sensor 115 receiving light for each image frame, image sensor 115 outputs the image frame to device 101 (e.g., to one or more processors of device 101). Device 101 receives image frames from image sensor 115 and processes the image frames via one or more processors. Camera 105 can be part of device 101 or can be separate from device 101. In some implementations, camera 105 can include camera controller 125.

[0054] Figure 1The device 101 may include one or more processors. The one or more processors of the device 101 may include a camera controller 125, an image signal processor (ISP) 120, a processor 135, a digital signal processor (DSP) 130, or a combination of the above. The ISP 120 and / or the DSP 130 may process image frames from the image sensor 115. In some examples, the DSP 130 may be a host processor (HP) (also referred to as an application processor (AP) in some cases). The DSP 130 (as the HP) may be used to dynamically configure the image sensor 115 with new parameter settings. The DSP 130 (as the HP) may also be used to dynamically configure the parameter settings of the ISP 120 (e.g., to match the settings of the image frame from the image sensor 115 so that the image data is processed correctly).

[0055] ISP 120 and / or DSP 130 may generate visual media, which may be encoded using an image and / or video encoder. The visual media may include one or more processed still images and / or one or more videos including video frames based on image frames from image sensor 115. Device 101 may store the visual media as one or more files on memory 140. Memory 140 may include one or more non-transitory computer-readable storage media components, each of which may be a reference Fig.16 The memory 1615 of the present invention may be any type of memory or non-transitory computer-readable storage medium discussed above. In some cases, one or more of the one or more non-transitory computer-readable storage medium components of the memory 140 may be removable. Illustrative examples of the memory 140 may include a secure digital (SD) card, a micro SD card, a flash memory component, a hard drive, a random access memory (RAM) such as a dynamic RAM (DRAM) or a static RAM (SRAM), another storage medium, or some combination of the above.

[0056] The display 150 may be any suitable display or screen that allows user interaction and / or is used to present items (such as captured image frames, videos, or preview images) for viewing by the user. In some aspects, the display 150 may be a touch-sensitive display. The I / O component 155 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from a user and provide output to the user. For example, the I / O component 155 may include (but is not limited to) a graphical user interface, a keyboard, a mouse, a microphone, and a speaker, etc. The display 150 and / or the I / O component 155 may provide a preview image to the user and / or receive user input for adjusting one or more settings of the camera 105 and / or the ISP 120 (such as selecting and / or deselecting a region of interest of a displayed preview image for autofocus (AF) operation).

[0057] ISP 120 can process captured image frames or videos provided by image sensor 115 of camera 105. ISP 120 can include a single ISP or can include multiple ISPs. Examples of tasks that can be performed by different modules or processing blocks of ISP 120 can include demosaicing (e.g., interpolation), autofocus (and other automatic functions), noise reduction (also known as denoising or noise filtering), lens / sensor noise correction, edge enhancement (e.g., using a sharpening filter), color balance, contrast, intensity adjustment (e.g., dimming or brightening), hue adjustment, Bayer filtering (using Bayer filtering), color conversion, correction or enhancement / suppression of image attributes, and / or other tasks. In some examples, camera controller 125 (e.g., ISP 120) can also control the operation of camera 105. In some cases, ISP 120 can process received image frames using parameters provided by a parameter database (not shown) stored in memory 140. Processor 135 can determine the parameters to be used by ISP 120 from the parameter database. ISP 120 may execute instructions from a memory (e.g., memory 140) to process image frames or video, may include specific hardware to process image frames or video, or may additionally or alternatively include a combination of specific hardware and the ability to execute software instructions for processing image frames or video.

[0058] In some examples, image frames may be received by device 101 from sources other than a camera, such as other devices, equipment, network attached storage and / or other storage, and other sources. In some cases, device 101 may be a test device in which ISP 120 is removable such that another ISP may be coupled to device 101 (such as a test device, test equipment, etc.).

[0059] Components of device 101 may include, and / or may be implemented using, electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (DPU), and / or other suitable electronic circuits), and / or may include, and / or be implemented using computer software, firmware, or any combination of the foregoing, to perform the various operations described herein.

[0060] Although device 101 is shown as including specific components, one of ordinary skill in the art will appreciate that device 101 may include more than Figure 1 More or fewer components than shown. For example, device 101 may also include one or more input devices and one or more output devices (not shown). In some implementations, device 101 may also include or be part of a computing device that includes one or more memory devices other than memory 140 (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and / or other memory devices), a processing device other than processor 135 and / or DSP 130 that communicates with and / or is electrically connected to the one or more memory devices (e.g., one or more CPUs, GPUs, and / or other processing devices), one or more wireless interfaces for performing wireless communications (e.g., including one or more transceivers and a baseband processor for each wireless interface), one or more wired interfaces for performing communications over one or more hardwired connections (e.g., serial interfaces (such as universal serial bus (USB) inputs, lightning connectors), and / or other wired interfaces), and / or Figure 1 Other components not shown.

[0061] In some implementations, device 101 may include a camera device, a mobile device, a personal computer, a tablet computer, a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, and / or a mixed reality (MR) device), a server (e.g., in a software as a service (SaaS) system or other server-based system), and / or any other computing device having the resource capabilities to perform the techniques described herein.

[0062] In some cases, device 101 may include one or more software applications, such as a camera debugging application that incorporates the techniques described herein. The software application may be a mobile application, a desktop application, or other software application installed on device 101.

[0063] As described above, the camera system or components of the camera system (e.g., ISP 120, image sensor 115, and / or other components) can be debugged so that the camera system provides the desired image quality. For example, the parameters of ISP 120 can be adjusted to optimize the performance of ISP 120 when processing image frames captured by image sensor 115. In some cases, the image quality system and / or software can analyze the image frames (e.g., digital images and / or image frames) output by the camera system (e.g., camera system 100). For example, the image quality system and / or software can analyze the image frames using one or more test charts (such as TE42 charts, etc.). The image quality system and / or software can output a variety of image quality (IQ) metrics about the characteristics of the camera system. IQ metrics may include characteristics such as Opto-Electric Conversion Function (OECF), dynamic range, white balance, noise and ISO speed, visual noise, Modulation Transfer Function (MTF), limiting resolution, distortion, lateral and / or longitudinal chromatic aberration, vignetting, shadows, glare, color reproduction, combinations of the above, and / or other characteristics.

[0064] The characteristics of the camera system can be used to perform various functions for debugging the camera system. For example, image quality issues can be debugged and ISP parameters can be fine-tuned based on specific user IQ requirements. In some cases, users can include original equipment manufacturers (OEMs). Different OEMs can have different quality requirements for different devices. For example, based on the quality requirements of a specific OEM and the characteristics provided by the image quality system and / or software, the ISP parameters can be adjusted so that the performance of the ISP is optimized when a certain task is used to process communication frames. As described above, the tasks of the ISP can include demosaicing (e.g., interpolation), autofocus (and other automated functions), noise reduction, lens correction, and other tasks.

[0065] As described above, camera debugging can be a time-consuming and resource-intensive process. For example, debugging the parameters of the ISP of a camera system may require a rigorous manual process, which in some cases takes weeks to complete. The initial part of the camera debugging process may include coarse tuning of the parameters of the ISP. Coarse tuning of the parameters of the ISP may include debugging the parameters with a benchmark IQ as the target. In an illustrative example, if the camera system of a device (e.g., a mobile phone) of a particular OEM has the best IQ on the market, that IQ can be used as a benchmark IQ. When debugging other devices on the market, the debugging engineer can get as close to the benchmark IQ as possible in the first round of debugging of the other devices. An example of a benchmark for IQ evaluation is DXOMark (https: / / www.dxomark.com), for example, based on DXOMark Analyzer.

[0066] As the number of adjustable parameters for an ISP may reach hundreds or thousands, a reduced number of IQ metrics can be mapped to the adjustable parameters of the ISP. Mapping the reduced number of IQ metrics to adjustable parameters can enable people debugging the ISP to focus on these reduced number of IQ metrics rather than a larger number of adjustable parameters. An IQ metric is a measurement of perceptible attributes of an image (each perceptible attribute is called "ness"). Example attributes or ness include the brightness of an image frame, the clarity of an image frame, the granularity of an image frame, the hue of an image frame, the color saturation of an image frame, and the like. These attributes or ness, once changed for a particular image frame, are perceived by a person. For example, if the brightness of an image frame is reduced, a person will perceive the image frame as darker.

[0067] In some examples, the number of IQ metrics can be 10-20 (or other number), where each IQ metric corresponds to multiple adjustable parameters. In some cases, two or more different IQ metrics can affect some of the same adjustable parameters for the ISP. In some examples, the parameter database can associate different values ​​of the IQ metric with different values ​​of the parameter. For example, an input vector of IQ metrics can be associated with an output vector of adjustable parameters so that the ISP can be adjusted for the corresponding IQ metric. Because the number of parameters can be large, the parameter database may not store all combinations of IQ metrics, but may include a portion of the number of combinations. In Figure 1 Where device 101 is depicted as having a parameter database in memory 140, the database may be stored external to device 101 (such as in a network attached storage, cloud storage, a test device coupled to device 101, etc.). In some cases, these parameters may affect components external to the ISP (such as Figure 1105). The present disclosure should not be limited to the specifically described parameters or to parameters that are specific to only an ISP. For example, these parameters may be used for a specific ISP and camera (or image / camera sensor) combination, or for different ISP and camera (or image / camera sensor) combinations.

[0068] In some examples, an IQ model can be used to map an IQ metric to an adjustable parameter. Any type of IQ model can be used, and the present disclosure is not limited to a particular IQ model that associates an IQ metric with an ISP parameter. In some examples, the IQ model can include one or more modulation transfer functions (MTFs) to determine changes in ISP parameters associated with changes in the IQ metric. For example, changing the brightness IQ metric can correspond to parameters associated with adjusting image / camera sensor sensitivity, shutter speed, flash, determining the ISP for the intensity of each pixel of the input image, adjusting the ISP for the hue or color balance of each pixel for compensation, and / or other parameters. The brightness MTF can be used to indicate that a change in the brightness IQ metric corresponds to a particular change in the associated parameter.

[0069] The IQ model and / or MTF may vary between different ISPs, or between different combinations of ISPs and cameras (or camera / image sensors). Tuning the ISP may include determining differences between MTFs or in the IQ model such that the IQ metric value is associated with a preferred adjustable parameter value for the ISP (in a parameter database). The "optimal" processed image frame may be based on user preferences, or may be subjective to one or more experts, such that optimization of the IQ model is open ended and subject to differences between users or between people assisting in the tuning. However, the IQ may be quantified, such as by using an IQ range (such as from 0 to 100, with 100 being the best) to indicate the IQ performance for the ISP and / or camera. For example, the IQ for a processed image frame may be quantified, and the expert may use the quantification to tune the ISP (such as to adjust or determine parameters of the ISP or a combination of the ISP and camera sensor).

[0070] Some IQ metrics may be relative to each other, such as noisiness (corresponding to the amount of noise) and texture, where reducing or increasing noise may correspondingly reduce or increase high frequency texture information in the image. When tuning an ISP, tradeoffs are determined between IQ metrics in an attempt to optimize the processing of the image (e.g., by generating the highest quantized IQ score from the IQ range).

[0071] Optimizing the IQ metric or otherwise tuning the ISP may be different for different scene types. For example, an indoor scene illuminated by incandescent lamps may correspond to a different "optimal" IQ metric (and corresponding parameters) than an outdoor scene with bright natural lighting. In another example, a scene with a flat area of ​​large color and brightness may correspond to a different "optimal" IQ metric than a scene with a large amount of color and color variation in the area. As a result, the ISP can be tuned for a variety of different scene types.

[0072] The goal of camera tuning is to achieve better IQ than previously available products on the market. As image / camera sensors and ISPs continue to evolve, this improvement in IQ will become more prominent. As described above, the coarse tuning of ISP parameters can target a baseline IQ. In some cases, it is difficult to achieve the same IQ as the baseline IQ. For example, different devices (e.g., different mobile device camera systems) may have different image / camera sensor and ISP combinations and / or configurations. Such differences can make it difficult for the device to achieve and in some cases impossible to achieve the same type of compromise as the device that sets the baseline IQ. After the initial coarse tuning, fine tuning (e.g., user preference tuning) can be performed. For example, a user (e.g., an OEM) can provide specific feedback (e.g., requirements), which can be implemented by a debugging engineer who is fine-tuning the image / camera sensor and / or ISP of a particular device. Examples of feedback can include requests for more noise removal (e.g., denoising) under one or more low-light conditions (e.g., low light, normal light, bright light, and / or other illumination conditions), better saturation levels in bright light, and / or other feedback.

[0073] Figure 2 2 is a diagram illustrating an example of a manual tuning process 200 for tuning ISP parameters. The manual tuning process 200 is performed to determine how ISP parameter changes reflect image quality. At operation 202, the process 200 includes modifying the ISP parameters (e.g., based on feedback received from a previous iteration of operation 202). At operation 204, the process 200 includes performing a camera simulation using the currently tuned ISP parameters (e.g., as modified at operation 202). The result of operation 204 may be one or more output images and / or one or more output video frames.

[0074] At operation 206, process 200 includes performing a subjective visual assessment of one or more output images and / or video frames to determine whether a desired change in the output has occurred. Process 200 is repeated by providing feedback based on the subjective visual assessment performed at operation 206. In some cases, a designer and / or manufacturer of a camera system may perform operations 202, 204, and 206 based on requirements provided by a user (e.g., an OEM). In some cases, a designer and / or manufacturer of a camera system may perform operations 202 and 204, while a user (e.g., an OEM) may perform operation 206.

[0075] Figure 3A and Figure 3B is an example of image frame 302 and image frame 304 captured by a camera of a mobile device having a sensor-ISP combination of an IMX363 sensor and a Snapdragon 845 ISP. Figure 3A and Figure 3B Frames 302 and 304 in FIG. 3 show the expected IQ change resulting from fine-tuning (from the debugger's perspective). In particular, Figure 3A and Figure 3B Image frame 302 and image frame 304 in FIG. 3 provide a comparison between the coarsely tuned settings and the finely tuned settings for texture and noise. Figure 3A The image frame 302 in is generated by an ISP with coarsely tuned settings. Figure 3B The image frame 304 in FIG. 3 is generated by an ISP with fine-tuned settings. Figure 3A and Figure 3B It can be seen that compared with the image frame generated using the coarse settings ( Figure 3A Compared with the image frame 302 in FIG. 1 , the fine-tuned setting is the same image frame ( Figure 3B Image frame 304 in FIG. 3 provides improved texture detail and clearer noise distribution.

[0076] Figure 4A and Figure 4B is another example of image frame 402 and image frame 404 captured by a mobile device having a sensor-ISP combination of an IMX363 sensor and a Snapdragon 845 ISP. Figure 4A and Figure 4B Frames 402 and 404 in FIG. 4 show the expected IQ change resulting from fine-tuning (from the debugger's perspective). In particular, Figure 4A and Figure 4B Image frame 402 and image frame 404 in FIG. 4 provide a comparison between the coarsely adjusted settings and the finely adjusted settings for resolution. Figure 4A The image frame 402 in FIG. 4 is generated by an ISP having coarsely adjusted settings, while Figure 4BThe image frame 404 in is generated by an ISP with fine-tuned settings. Figure 4A and Figure 4B It can be seen that compared with the image frame generated using the coarse settings ( Figure 4A Compared with the image frame 402 in FIG. 4 , the fine-tuned setting is the same image frame ( Figure 4B The image frame 404 in FIG. 4 provides a higher frequency resolution.

[0077] Figure 5A and Figure 5B are examples of image frames 502 and 504 captured by a camera. Image frames 502 and 504 illustrate the expected IQ change (from the end user's perspective) resulting from fine-tuning. Figure 5A The image frame 502 in FIG. 5 is generated based on the default ISP parameter settings selected by one or more commissioning engineers. However, the camera end user may have a fixed preference for a 6 degree hue shift and a 9% saturation increase for skin tones, such as Figure 5B As shown in the image frame 504 in FIG. 1 , the remaining quality aspects of the image frame 504 are maintained as Figure 5A The same as the image frame 502. Currently, the camera end user cannot change the ISP parameter settings to make the changes he wants.

[0078] A manual iterative process for evaluating how small ISP parameter changes are reflected in the image quality (IQ) of the image frame (e.g. Figure 2 The process shown in ( ) is lengthy and inefficient for multiple ISP-sensor combinations. The process becomes even more lengthy and inefficient when performed under different operating conditions (e.g., the same parameters are tuned for different illumination conditions). In some cases, even after the tuning engineer finally determines the ISP parameters for the optimal IQ (e.g., referring to the OEM's preferences), the image quality may not be consistent with the ideal or expected IQ based on the perspective of the camera end user.

[0079] As described above, existing camera debugging systems and techniques require debugging engineers to have expertise in many ISP parameters (e.g., thousands of ISP parameters). This expertise is required to manually debug the ISP parameters based on feedback from users (e.g., OEMs) to obtain a suitable IQ tradeoff (e.g., texture-noise tradeoff). The above iterative manual process (e.g., reference Figure 2) is a time and resource intensive process, including simulation and visual evaluation after each small change in parameter settings. For example, manual debugging for eight different illumination conditions may take 7-10 days, or even longer in some cases. Such a process is inefficient and has repeatability issues. It is also tedious to track every slight change in parameter settings from beginning to end. In addition, every time a new ISP module is added or modified, the debugging engineer needs to have a deep understanding of the impact of all parameters of the new ISP module on the IQ of the output image frame.

[0080] As described above, automatic camera debugging systems and techniques are described herein that provide automatic camera debugging. For example, the automatic camera debugging system and techniques can be used to automatically debug an ISP, a camera sensor (or image sensor), or other components of a camera system. In some examples, an automatic camera debugging tool can be used to implement automatic camera debugging. For example, any type of user (e.g., an OEM debugging engineer, a camera end user, and / or other user) can perform fine-tuning of the ISP by interacting with a graphical user interface (GUI) of the automatic camera debugging tool.

[0081] The GUI of the camera debugging tool may include selectable graphical elements. The user may interact with the selectable graphical elements to indicate the image quality (IQ) change desired by the user. For example, based on the user's selection of one or more selectable graphical elements, the camera debugging tool may perform a real-time (or near real-time) selection of ISP parameter settings with reference to the IQ change desired by the user. In some cases, using the GUI of the camera debugging tool, the user may select a specific coarsely adjusted setting and may guide the type of IQ improvement desired or required relative to the coarsely adjusted setting (e.g., increase in texture, reduction in noise, etc.). The automatic camera debugging tool may generate a new setting option with an overall IQ similar to the selected setting, wherein the desired aspects of the IQ are enhanced. In some cases, the camera debugging tool and / or the GUI of the camera debugging tool may be different for different types of users. For example, a first GUI may be provided to an OEM user, and a second GUI (different from the first GUI) may be provided to a camera terminal user.

[0082] In some cases, the GUI of the automatic camera commissioning tool (e.g. Figure 6) can be used to obtain user feedback on specific aspects of IQ (e.g., texture, noise, edge definition, ringing effects, etc.). The feedback can be transformed or converted into a corresponding IQ metric target (e.g., by determining the target metric value using equation (2) below). In some cases, a parameter setting search can be performed to search from pre-generated compromise ISP settings to obtain a setting that provides an IQ metric that is closest to the IQ metric target. A compromise ISP setting refers to a set of data points with different IQ metrics (e.g., points with high texture-high noise and low texture-low noise). For example, when debugging, a user (e.g., an OEM) can modify parameters to obtain an optimal "tradeoff" between a noise metric and a texture metric. In some cases, the camera debugger can pre-generate multiple ISP settings, each ISP setting having a different tradeoff (e.g., a texture-noise tradeoff) from which a user (e.g., an OEM) can select. Each ISP setting corresponds to an IQ metric tradeoff.

[0083] For example, as described above and in detail below, a parameter setting search may be performed to identify a specific set of settings that meet an IQ metric target. Using the automatic camera tuning tool, a user may select a specific IQ metric (also referred to as an IQ feature) that the user desires to adjust for a given setting (e.g., an ISP setting), and a parameter setting search may be performed to determine the specific settings that correspond to the user's selection. In one illustrative example, for a given ISP setting, a user may indicate a desire to reduce the noise of image frames produced using the given ISP setting. A parameter setting search may be performed to determine the optimal ISP setting that will provide the desired noise quality, but will not reduce the quality of other IQ metrics (e.g., texture, resolution, etc.). In some cases, the user may indicate the strength of the IQ metric adjustment (e.g., reduce by a factor of -1, -2, -3, etc., or increase by a factor of 1, 2, 3, etc.).

[0084] Figure 6 6 is a diagram showing an example of a graphical user interface (GUI) 600 of an automatic camera debugging tool. An example of a user of GUI 600 is an OEM user (e.g., a debugging engineer, a device engineer, a software engineer, or other user) who can debug the ISP of a device being manufactured by the OEM. Another example of a user of GUI 600 is an end user (e.g., a consumer who purchases a camera that can be debugged using the automatic camera debugging tool). Figure 6As shown, a plurality of settings are shown in an image quality (IQ) metric table 601. Each setting is included in the IQ metric table 601 with a given setting number, including setting numbers 0, 1, 2, 3, 0, N-0, 0, N-1, and 0, N-3. Each setting corresponds to a tuned ISP parameter with which the ISP has been tuned. For example, the settings in the IQ metric table 601 may include a coarsely tuned ISP metric that may be fine-tuned using an automatic camera tuning tool based on input received via the GUI 600.

[0085] For each ISP setting, values ​​for a variety of IQ metrics are displayed in the IQ metric table 601, including a noise metric, a texture metric, and a resolution metric. For example, for the ISP setting with setting number 0, the value of the noise metric is 83.95, the value of the texture metric is 84.91, and the value of the resolution metric is 85.19. The values ​​provided for the IQ metrics may include any value indicating the quality of a given metric (e.g., any score-based value). In some examples, Figure 6 The example values ​​of the IQ metrics shown in can be generated using a scoring mechanism that combines multiple IQ metrics to provide a score for a given feature. For example, multiple IQ metrics can correspond to different aspects of clarity, including an IQ metric for texture in a high contrast area, an IQ metric for texture in a low contrast area, and an IQ metric for resolution. The IQ metrics can be combined together to generate a clarity score representing clarity. In another example, noise in the brightness domain and noise in the color domain can be combined into a noise score. This score generated using multiple IQ metrics can be referred to as an IQ score. Figure 6 Also shown is an IQ metric chart 603 . The IQ metric chart 603 plots different IQ metric values ​​for different settings of the IQ metric table 601 .

[0086] GUI 600 includes a variety of selectable graphical elements that a user can interact with to operate the automatic camera tuning tool. For example, setting number graphical element 602 allows a user to select a specific setting number to make fine adjustments. Tuning options graphical element 604 allows a user to select a user-preferred tuning option to adjust the setting selected using setting number graphical element 602. Figure 6 6, the user has selected "reduce noise" as the preferred adjustment for setting number 0. Strength bar 606 is provided as a selectable graphical element to allow the user to indicate the strength or intensity of the adjustment to be applied to the tuning option (e.g., noise) of the selected setting (e.g., setting number 0). Strength bar 606 is optional and can be omitted from the tuning tool GUI 600 in some implementations. The user can select the start fine tuning graphical element 608 to cause the automatic camera tuning tool to begin the fine tuning process.

[0087] Figure 7 is a diagram showing a GUI for a camera based on an automatic camera commissioning tool (e.g., Figure 6 600). Process 700 can be used to fine-tune camera settings (e.g., ISP settings) based on user preferences, as indicated by using a GUI (e.g., GUI 600) of an automatic camera commissioning tool.

[0088] At operation 702, process 700 includes receiving an indication of selecting a coarse setting for an ISP or other camera component. For example, in response to a user selecting Figure 6 600 of the GUI 600, the process 700 may receive an indication of selecting a coarsely adjusted setting. The coarsely adjusted setting may be based on tuning of the ISP (or other camera components) to achieve a baseline IQ. The user may select a setting based on the displayed IQ score (e.g., Figure 6 The user may select a setting by selecting an IQ score (shown in IQ metric table 601). As described herein, an IQ score may be determined for specific IQ characteristics (e.g., clarity, noise, artifacts, etc.) and correlated to a subjective IQ. IQ may be calculated for each ISP setting and displayed to the user to help select a setting for fine tuning.

[0089] The user may wish to fine-tune the coarsely adjusted settings based on one or more IQ metrics. The user may select one or more graphical elements of the GUI to cause the automatic camera commissioning tool to adjust the one or more IQ metrics of the coarsely adjusted settings. At operation 704, process 700 includes receiving an indication of selecting an IQ metric for adjustment. For example, process 700 may be responsive to a user selecting (e.g., using Figure 6 600) to receive an indication of the IQ metric to be adjusted for a particular setting. A particular setting. As described above, a user may select a particular setting using setting number graphical element 602. A variety of IQ metrics may be selected for adjustment, including noise, sharpness, texture, edge, overshoot, resolution, etc.

[0090] At operation 706, process 700 includes receiving an indication of a selected intensity of adjustment. For example, process 700 may be responsive to a user selecting (e.g., using Figure 6 The intensity bar 606 in the GUI 600 of the embodiment of the present invention can be used to receive an indication of selecting the intensity of adjustment by indicating the intensity or strength of the adjustment of the IQ metric. In one illustrative example, the user can indicate that the noise is to be reduced by a factor of -2.

[0091] At operation 708, process 700 includes generating new settings with updated IQ scores based on the selections of operation 702, operation 704, and operation 706. In one illustrative example, the new settings with updated IQ scores may be displayed on Figure 6 In some implementations, the parameter setting search process (hereinafter referred to as Figure 8 The ) can be used to generate new settings based on a user's selection of a setting, an IQ metric for adjustment, and an optional adjustment strength.

[0092] At operation 710, process 700 includes determining whether an indication to select additional settings is received. As described above, a user may select a setting based on a displayed IQ score (e.g., Figure 6 The additional setting may include another coarsely adjusted setting or a finely adjusted setting after the coarsely adjusted setting has been updated due to the user selecting the setting for adjustment. If selection of an additional setting is determined, the process 700 returns to operation 704 to receive an indication of selecting an IQ feature to adjust the additional setting. In some cases, the process 700 may be repeated until no setting is selected.

[0093] In some cases, once no additional settings are selected, process 700 proceeds to operation 712. At operation 712, process 700 includes providing an option to simulate the final settings. A simulation of the final settings may be performed for user verification and / or comparison. For example, a GUI for an automatic camera commissioning tool may provide a simulation option (e.g., Figure 6 The simulation option allows the user to simulate any setting for direct visual assessment (e.g., by displaying a comparison graphic element 610 of the GUI 600). Figure 6 In some examples, the automatic camera tool may provide image frames generated using multiple settings for the user to compare (e.g., by displaying a first image frame generated using a setting with setting number 0 and a second image frame generated using a setting with setting number 1).

[0094] As described above, user feedback on specific aspects of IQ (e.g., texture, noise, edge sharpness, ringing, resolution, etc.) can be obtained and converted into corresponding IQ metric targets. A parameter setting search process can be performed to search through pre-generated tradeoff settings to obtain settings that produce the desired metric targets. The parameter setting search process can be based on the GUI of the automatic camera commissioning tool (e.g., Figure 6The points in a database (or other storage mechanism) may be operated on based on user feedback provided by the GUI 600). For example, when coarse tuning an ISP or other component of a camera system (e.g., using the SmartU2 coarse tuning tool), a dense database of points may be created. Each data point in the database may correspond to a particular coarsely tuned ISP parameter setting. In some cases, each data point may be stored (e.g., as a tuple or other data structure) with an IQ metric and an ISP parameter setting. Based on the user's feedback, the points in the database may be searched for optimal points (e.g., corresponding to optimal tuned ISP parameter settings).

[0095] Each data point can be marked in the database by a set of IQ metrics and scores. The IQ metrics may include standardized metrics for global IQ evaluation. In some examples, the IQ metrics may include calculations based on visual noise metrics and modulation transfer functions (MTF) for features such as texture, resolution, edge clarity, and / or other features. In some examples, the IQ metrics may be calculated using a TE42 chart, which is a multi-purpose chart for camera testing and debugging. The TE42 chart has multiple parts and can be used to measure the photoelectric conversion function (OECF), dynamic range, color reproduction quality, white balance, noise, resolution, shadows, distortion, and kurtosis of a camera system. In addition to or as a replacement for the TE42 chart, one or more other charts, such as QA-62 charts, TE106 charts, and / or other charts, may also be used. In some cases, as described above, a score may be obtained by combining multiple IQ metrics. In an illustrative example, a clarity score may be determined by combining (e.g., using an addition formula) the MTFs for high-frequency resolution, low-frequency resolution, high-contrast texture, and low-contrast texture. In another illustrative example, a noise score may be determined by combining the luminance and chrominance aspects of visual noise.Other scores for data points may also be determined.

[0096] In some examples, the IQ scores provided for points in the database may be for clarity and noise, and / or for other characteristics. These scores are based on IQ metrics, and user (e.g., OEM) engineers and debuggers can rely on these scores to provide accurate correlation with the subjective image quality of the image frames produced by the ISP. Thus, the scores may provide useful screening criteria.

[0097] Figure 8 8 is a flow chart illustrating an example of a parameter setting search process 800 for performing fine tuning of ISP parameters. The process 800 can be used to transform or convert user feedback into corresponding IQ metric targets (also referred to as target metric values). The process 800 can also be used to search a database of settings to obtain settings that provide a desired metric target.

[0098] At operation 802, process 800 includes receiving an indication of a selected setting and an IQ metric to be adjusted. Determining the selected setting and the IQ metric to be adjusted may be based on Figure 7 Operation 702, operation 704 and operation 706 of process 700. For example, a user may use Figure 6 The user may select a setting using setting number graphical element 602 of GUI 600. The user may select the IQ metric of the setting to be adjusted using debug options graphical element 604 of GUI 600. In some examples, as described above, the user may select the strength or intensity of the adjustment (e.g., strength bar 606 of GUI 600).

[0099] At operation 804, process 800 includes removing points with redundant metric values ​​for the selected IQ metric and / or points with an IQ score worse than the selected setting. For example, since the user can fine-tune the camera debugging tool with different settings as the starting point, it is possible to reach the same data point via multiple paths. For example, a request to reduce noise on Setting_0 and a request to reduce texture on Setting_0 may cause the camera debugging tool to output the same setting. Operation 804 can be performed to remove redundant points so that if the output of the current fine-tuning step for the user already exists in the IQ metric table (e.g., as a result of a previous fine-tuning step), another copy of the output will not be added to the table. In this example, if the setting of the data point has the same metric as another setting already displayed on the IQ metric table, the setting will not be displayed in the IQ metric table as a new setting. In some cases, operation 804 can be performed to remove points with an IQ score worse than the selected setting, because a low IQ score can represent a bad data point. Operation 804 is optional and may not be performed in some implementations.

[0100] At operation 806, process 800 includes determining whether the selection of the IQ metric indicates an increase or decrease in the IQ metric. For example, as described above, debug options graphical element 604 allows a user to indicate which IQ metric is to be adjusted and how it is to be adjusted (e.g., to increase the IQ metric or to decrease the IQ metric). Process 800 may perform different operations based on whether the IQ metric is to be increased or decreased. For example, if the IQ metric is to be decreased, process 800 may perform operation 808, and if the IQ metric is to be increased, operation 812 may be performed.

[0101] At operation 808 (when the IQ metric is to be reduced), process 800 includes removing from the current search all points having a larger metric value of the selected IQ metric than the metric value of the selected setting's IQ metric. In one illustrative example, the selected IQ metric may include noise, and the noise value of the selected setting may be 82. In this example, any point (corresponding to the parameter setting) having a noise value greater than 82 may be removed from the current search. Operation 808 may be performed to prune the data points so that fewer data points are searched. Thus, the pruning performed by operation 808 may result in a more efficient search process. At operation 810, process 800 includes arranging or arranging the points in descending order so that the values ​​are listed from largest to smallest. For example, the points may be arranged in descending order relative to a particular IQ metric that the user requests to be enhanced. For example, a user may request an automatic camera commissioning tool to increase resolution or increase texture, in which case the points may be sorted in descending order based on resolution or texture (corresponding IQ metric).

[0102] At operation 812 (when the IQ metric is to be increased), process 800 includes removing from the current search all points having a smaller metric value of the selected IQ metric than the metric value of the selected setting's IQ metric. In one illustrative example, the selected IQ metric may include resolution, and the resolution value of the selected setting may be 85. Any point (corresponding to the parameter setting) having a resolution value less than 85 may be removed from the current search. Similar to operation 808, operation 812 may be performed to prune the data points so that fewer data points are searched. At operation 814, process 800 includes arranging or arranging the points in ascending order so that the values ​​are listed from smallest to largest.

[0103] At operation 816, process 800 includes determining a metric factor. As described below, the metric factor can be used in operation 818 to determine a target metric value. The metric factor can be determined based on the selected IQ metric and the data points having extreme values ​​(extreme values) of the IQ metric among the data points remaining after the pruning operation of operation 808 or operation 812. The extreme value can be the data point with the smallest or largest IQ metric value among the remaining data points. In some cases, the total size of the database (e.g., the number of data points) can also be considered when determining the metric factor. In some examples, the total size of the database can include the size of the entire database (before operation 804 and operation 808 or operation 812 are performed). In some examples, the total size of the database can include the size of the database after operation 804 and operation 808 or operation 812 are performed. In one illustrative example, the metric factor can be determined or calculated as follows (based on the total size of the database):

[0104]

[0105] Among them, multfact is the measurement factor, metric current is the value of the selected metric, metric extrema is the value of the extreme data point, and total size of database is the size of the database (before or after operation 804 and operation 808 or operation 812 are performed). Equation (1) provides the distance between each point in the database (or in some cases less than each point in the database), assuming that there is a uniform distribution in the database (e.g., the distance between the current point and the extreme value divided by the total number of points). The multfact term indicates the step size from the current metric to the extreme value metric, assuming that the data points in the database are uniformly distributed. As indicated below with reference to operation 818, the intensity of the adjustment indicated by the user (e.g., selected using intensity bar 606) can be used to determine how many steps are required based on the step size indicated by multfact.

[0106] At operation 818, process 800 includes determining a target metric value. The target metric value may be determined based on the selected IQ metric, the strength or intensity indicated by the user (e.g., selected using strength bar 606), the desired amount of output (e.g., how much output to provide), and a metric factor (e.g., multfact). In one illustrative example, the target metric value may be determined or calculated as follows:

[0107] metric target =metric current +(strength)*(idx of output)*(multfact)

[0108] Equation (2)

[0109] Among them, metric target is the target metric, current is the value of the selected metric, strength is the strength or intensity of adjusting the IQ metric (e.g., selected by the user using strength bar 606), idx of output is the index of the output according to how many outputs (or number of outputs) are to be provided, and multfact is the metric factor determined using equation (1). As shown in equation (2), the target metric (metric target ) is a step size defined based on the metric factor (multfact) for the selected IQ metric (metric current) is modified. The number of steps is controlled by the strength of the adjustment and the number of outputs (idx of output). For example, if the user indicates that the noise is expected to be reduced by a factor of -2 (where strength = -2), then the current metric (metric current ) minus twice the step size defined by multfact, resulting in a greater noise reduction than a force of -1 or 0.

[0110] If multiple outputs are desired, each output is generated using an incremental value based on the number of multiple outputs. For example, if the user indicates that two outputs are desired, two target metrics may be determined. For the first target metric, the index of the output may be equal to 1, which corresponds to the first step defined by the strength value and the metric factor value. For the second target metric, the index of the output may be equal to 2, which corresponds to the second step defined by the strength value and the metric factor value. In one illustrative example, if the user indicates that the noise is desired to be

[0111] If the noise is reduced by a factor of 1 (strength = -1) and the camera debugging tool is requested to generate two outputs, the first target metric (idx of output = 1) will be determined as the target metric minus the value of multfact (a single step due to the strength of 1). The second target metric (idx of output = 2) will be determined as twice the value of the target metric minus multfact (two steps). In another illustrative example, if the user indicates that the noise is desired to be reduced by a factor of -2 ((strength = -2) and two outputs are requested, the first target metric (where idx of output = 1) will be determined as twice the value of the target metric minus multfact (two steps due to the strength of 2). The second target metric (where idx of output = 2) will be determined as four times the value of the target metric minus multfact (four steps).

[0112] At operation 820, process 800 includes outputting the data point having the IQ metric that is closest to the target metric value determined at operation 818. In some cases, the IQ score associated with the data point may also be output. For example, the IQ score associated with the output data point may be displayed on Figure 6The IQ metric table 601 of FIG. 601 is shown in FIG. 602. (For example, as shown in Tables 1 and 2 below). As described above, each data point has an associated debugged parameter setting (e.g., a debugged ISP parameter setting). Therefore, identifying the data point having the IQ metric value closest to the target metric value is essentially identifying the debugged parameter setting that achieves the enhancement or adjustment target indicated by the user feedback. For each index of the output corresponding to the number of outputs, a data point can be output. For example, in the example above where the user requests two outputs, for index value 1 (where idx of output = 1), a first data point can be output, and for index value 2 (where idx of output = 2), a second data point can be output.

[0113] Examples of the application of the above automatic camera debugging process refer to the following Tables 1 and 2 and Figures 9A-11B Table 1 shows the results of subjective IQ improvement in a camera. In the example of Table 1, a requirement may be indicated (e.g., from an OEM) that noise be reduced with a slight improvement in clarity and detail (or resolution). The user may choose (e.g., using Figure 6 GUI 600) has a coarse setting for the most desired clarity and resolution and can indicate (e.g., using Figure 6 GUI 600) The camera commissioning tool can reduce noise with strength-2 to obtain a first set of outputs, and then further reduce noise with strength-1 for the output setting Out_1_c, as shown in Table 1. The camera commissioning tool can perform process 700 and process 800 to generate the outputs shown in Table 1. As shown by the arrow from the sharpness score of 68.46 to the sharpness score of 72.12 in Table 1, both the sharpness score and the noise score have increased, indicating a better IQ. For the final output (further noise reduction with strength-1), only two filtered images Out_2_a and Out_2_b are simulated, and the one with better subjective sharpness (Out_2_a) is selected as the final output.

[0114]

[0115] Table 1: Scores for user-selectable fine-tuned settings displayed by the Camera Tuning Tool

[0116] The enhancement in noise and clarity obtained from the coarse-tuned setting compared to the final screened fine-tuned setting Out_2_a is given by Fig. 9A , Fig. 9B , Fig. 10A and Fig. 10B The image shown in is shown.

[0117] Fig. 9A and Fig. 9BImage frames 902 and 904 in FIG. 1 provide the noise distribution of the coarsely adjusted settings for the camera of a particular mobile device under 20 lux conditions (given by Fig. 9A ) and the noise distribution of the fine-tuned setting Out_2_a selected in Table 1 (given by Fig. 9B 904 in the image frame 904), as determined by the automatic camera tuning tool. It can be observed that the fine-tuned settings produce Fig. 9B The image frame 904 is produced by the coarse adjustment setting. Fig. 9A The image frame 902 produces less noise.

[0118] Fig. 10A and Fig. 10B Image frames 1002 and 1004 in FIG. 1 provide texture details of a coarsely adjusted setting for a camera of a particular mobile device under 20 lux conditions (given by Fig. 10A ) and the texture details of the fine-tuned settings Out_2_a selected in Table 1 (given by Fig. 10B ), as determined by the automatic camera tuning tool. It can be observed that the fine-tuning performed to obtain a clearer noise distribution did not harm the texture details. For example, Fig. 10B Although the noise of the image frame 1004 is reduced, its texture details are still the same as Fig. 10A The image frame 1002 is similar.

[0119] In some cases, a camera or device manufacturer (e.g., OEM) may require higher resolution capture to achieve better image quality. In this case, the image quality can be improved by manually iterating the simulation (e.g., using Figure 2 Fine-tuning the process 200 shown in ) can be very time consuming. For example, for a device with an IMX586 sensor and a Snapdragon 855 ISP sensor-ISP combination, a single simulation of 5-frame Multi-Frame Noise Reduction (MFNR) takes approximately 15 minutes on a high-performance computing device. The automatic camera tuning tool described herein can greatly reduce the time to obtain the desired fine-tuned enhancement. For medium light illumination conditions of the device, the coarse-tuned output has too much noise removal, resulting in a loss of detail. By using the automatic camera tuning tool, fine-tuning can be performed to restore the necessary texture detail. The coarse-tuned settings can be selected (e.g., from Figure 6 This fine-tuning is achieved by increasing the texture by +2 in IQ metric table 601) as shown in Table 2. As shown by the arrow from the clarity score of 72.9 to the clarity score of 74.50 in Table 2, the clarity is significantly improved, while the noise score is only slightly reduced.

[0120]

[0121] Table 2: Scores for user-selectable fine-tuned settings displayed by the Camera Tuning Tool

[0122] Fig.11A and Fig. 11B It shows how the Out_1_c settings obtained with the fine-tuning tool can be used to enhance texture details compared to the coarse-tuned settings without causing too much damage to the noise distribution. In particular, Fig.11A and Fig. 11B The images in provide images generated using coarsely tuned settings for a specific mobile device's camera ( Fig.11A ) and the image generated using the fine-tuned settings Out_1_c ( Fig. 11B ) as determined by the automatic camera tuning tool. It can be observed that the fine-tuned settings produce images with a higher level of texture detail (e.g. Fig. 11B Region 1104B vs. Fig.11A ), but the difference in noise distribution is not significant (as shown in the area 1104A). Fig. 11B Region 1102B vs. Fig.11A 1102A).

[0123] In some cases, as described above, the user of the automatic camera tuning tool may be an end user of a camera or a device (e.g., a mobile device) that includes a camera. For example, even given that the OEM provides well-tuned ISP settings in various products, end users may have their own preferences regarding the desired (subjective) image quality (IQ) of the output image frames. Existing end-user devices allow manual control of settings such as exposure, shutter speed, automatic white balance (AWB), etc. However, the manual control options (e.g., via GUI graphical elements) only cover some parts of the IQ, and some users may not be aware of how the controllable IQ metrics affect the image frames. Users typically have a better understanding of subjective image quality, such as sharpness, saturation, hue, etc.

[0124] Existing end-user devices can also perform post-processing functions using built-in filters and / or applications to achieve various effects on captured frames. However, such post-processing functions require additional repetitive work, especially if the user has a fixed type of preference for most captured image frames.

[0125] In some implementations, process 700 and search process 800 can be adapted to the end user in order to provide personalized customized camera settings suitable for a particular end user. For example, ISP parameters can be pre-set based on feedback from the end user indicating desired saturation levels, hues, sharpening, and / or other IQ metrics. In some examples, process 700 and process 800 can be performed during an initial camera settings setup process (e.g., when a user boots up a new mobile device), prompting the user to provide feedback on various IQ metrics. In some cases, Figure 8 Operation 802 of the search process 800 may be modified for use in an end-user based system. Fig.12 and Fig.13 As described, rather than selecting specific settings and IQ metrics, a user may select an image frame that displays characteristics that the user desires.

[0126] Fig.12 is a flow chart illustrating an example of a process 1200 for performing automatic camera commissioning based on end-user feedback. Fig.13 The process 1200 is described using an example graphical user interface (GUI) 1300 shown in FIG. 1202. At operation 1202, the process 1200 includes presenting to a user pre-existing captures (image frames) corresponding to different IQ metric tradeoffs. For example, referring to FIG. Fig.13 1300, a first image frame 1302, a second image frame 1304, and a third image frame 1306 are displayed in the GUI 1300. Image frames 1302, image frames 1304, and image frames 1306 are captures of natural scenes based on which IQ metrics can be calculated. In some examples, the displayed captures can be standard charts (e.g., TE42 charts, QA-62 charts, TE106 charts, and / or other charts) used to debug a camera. Image frames 1302, image frames 1304, and image frames 1306 can correspond to different saturation intensities of the same scene. For example, the first image frame 1302 can correspond to a saturation intensity of -1 (decreased saturation), the second image frame 1304 can correspond to a saturation intensity of 0 (no change in saturation), and the third image frame 1306 can correspond to a saturation intensity of +1 (increased saturation).

[0127] At operation 1204, process 1200 includes receiving one or more selections of one or more graphical elements for adjusting settings. Fig.13 As shown, GUI 1300 may include a scroll bar 1308. Scroll bar 1308 is a selectable graphical element that allows a user to select an image frame that corresponds to a user's desired saturation level. Fig.13A scroll bar 1308 is shown as an example of a selectable graphical element, other selectable graphical elements may also be used, such as a drop-down menu, a text input box, selectable image frames (e.g., the first image frame 1302, the second image frame 1304, and the third image frame 1306 may be selected by the user), any combination of the above, and / or other selectable graphical elements. Other selectable graphical elements may be displayed in association with other IQ metrics, such as selectable graphical elements (e.g., scroll bars) for selecting different hues, sharpness, saturation, and other IQ metrics in a capture. For example, graphical elements may be provided for increasing and / or decreasing saturation, sharpness, hue, and other IQ metrics to allow a user to select from different captures depicting different levels of IQ metrics.

[0128] At operation 1206, process 1200 includes converting one or more selections into corresponding target metrics and performing a search for an optimal output. Figure 8 The process 800 described may be used to perform the transformation of selections into target metrics and search databases of data points (corresponding to different ISP settings). For example, by using Figure 8 In process 800, user selections may be converted into corresponding target metrics (e.g., metric current ), and a search may be performed to determine the optimal output data point. As described above, the search may be performed among data points corresponding to pre-generated (e.g., generated offline) ISP settings. For example, the data points may include coarsely tuned ISP settings or pre-fine-tuned ISP settings.

[0129] At operation 1208, process 1200 includes loading the ISP settings corresponding to the optimal data point onto the ISP for future image capture. Figure 8 The ISP of the device is debugged using the ISP settings associated with the data points output at operation 820. Parameters corresponding to the newly searched settings loaded onto the ISP can be used for subsequent captures. As described above, each data point has been stored (e.g., as a tuple or other data structure) with an IQ metric and an ISP parameter setting, so these data points can be easily obtained and loaded onto the ISP. The search is performed in the metric space, and when a data point is selected based on the metric, the corresponding parameter settings are ready-made and can be quickly loaded onto the ISP. In some cases, if the user wants a different type of capture with different characteristics, the user can return to the GUI to re-debug the ISP settings.

[0130] Fig.14A and Fig. 14B1402 and 1404, showing the results of capturing using the original debugged settings of the device and using the Fig.12 For example, a user may select a preference for color saturation (e.g., using Fig.13 GUI 1300). Fig.14A The image frame 1402 corresponds to the original debugged ISP, while Fig. 14B Image frame 1404 corresponds to an ISP parameter setting selected to increase color saturation based on user feedback.

[0131] The automatic camera commissioning systems and techniques described herein provide advantages over existing camera commissioning techniques (e.g. Figure 2 For example, Figure 2 Compared to the manual debugging process 200 shown in , which takes 7-10 days, debugging for different illumination conditions (e.g., 8lux conditions) using the automatic camera debugging tool can take only one day using the automatic camera debugging system and technology described herein. In addition, each IQ change can be restored without tracking the underlying parameter changes, thereby ensuring easy repeatability. Another benefit is that only knowledge of subjective IQ is required, regardless of the evolution of the ISP. For example, the user does not need to learn the expertise of thousands of ISP parameters.

[0132] The automatic camera debugging systems and techniques described herein can internally transform subjective feedback from a user into a target metric, and accordingly, data points representing a compromise metric space are searched. ISP parameters corresponding to the data points selected by the metric target-based search can be output. This solution prevents the user from having to manually adjust thousands of parameters to obtain the desired IQ change. In some cases, from the end user's perspective, the techniques described herein provide the end user with control over personalized camera settings so that the desired processing of image frame capture is automatically obtained through pre-selected ISP settings. The desired image features can therefore be obtained without the need to use image post-processing.

[0133] The target metrics derived using the techniques described herein correlate well with the desired subjective IQ. The automated camera tuning tool generates tuned settings with enhanced subjective image quality as per user requirements with minimal simulation overhead and manual effort. The tool can reduce the fine-tuning time for 8 lighting conditions from 1 week to 2 days. The tool can also be integrated into existing camera tuning tools (e.g., Chromatix tuning tool).

[0134] Fig.1515 is a flow chart illustrating an example of a process 1500 for determining one or more camera settings using the techniques described herein. At block 1502, process 1500 includes receiving an indication of selecting an image quality metric for adjustment. In some examples, the indication of selecting the image quality metric includes a direction of adjustment. In some examples, the direction of adjustment includes a decrease in the image quality metric. In some examples, the direction of adjustment includes an increase in the image quality metric. In some implementations, the selection of the image quality metric for adjustment is based on a graphical element of a graphical user interface (e.g., Figure 6 In some aspects, the graphical element includes an option to increase or decrease the image quality metric. In some implementations, the graphical element is associated with a displayed image frame having an adjusted value of the image quality metric, such as Fig.13 In some implementations, the selection of the image quality metric for adjustment is based on the selection of the displayed image frame having the adjusted value of the image quality metric. Fig.13 , instead of operating scroll bar 1308 , the user may select one of image frame 1302 , image frame 1304 , or image frame 1306 to select an image quality metric for adjustment.

[0135] At block 1504, process 1500 includes determining a target image quality metric value for the selected image quality metric. In some examples, process 1500 includes determining a metric factor. In one example, operation 816 of process 800 may be performed to determine the metric factor. For example, process 1500 may determine the metric factor based on the metric value of the selected image quality metric, based on a data point in the plurality of data points having an extreme value of the selected image quality metric, and / or based on a number of the plurality of data points (e.g., as described above with reference to FIG. 1 ). Figure 8 Process 1500 may include determining a target image quality metric value for the selected image quality metric based on the metric factor and the metric value of the selected image quality metric (e.g., as described above with reference to Figure 8 818).

[0136] In some examples, process 1500 includes receiving an indication of selecting an adjustment level to be made to an image quality metric. For example, a user may use Figure 6 The process 1500 may include determining a target image quality metric for the selected image quality metric (e.g., as described above with reference to FIG. 1 ) based on the metric factor, the metric value of the selected image quality metric, and the intensity of the adjustment to the image quality metric. Figure 8 818).

[0137] In some examples, process 1500 includes receiving an indication of a number of desired output camera settings. For example, a user may use Figure 6 600 to select the desired number of output camera settings. Process 1500 may include determining a target image quality metric value for the selected image quality metric based on the metric factor, the metric value of the selected image quality metric, and the desired number of output camera settings (e.g., as described above with reference to FIG. 1 ). Figure 8 818).

[0138] In some examples, process 1500 includes receiving an indication of selecting a strength of adjustment to an image quality metric and receiving an indication of selecting a desired number of output camera settings. Process 1500 may include determining a target image quality metric value for the selected image quality metric (e.g., as described above with reference to FIG. 1 ) based on the metric factor, the metric value of the selected image quality metric, the strength of adjustment to the image quality metric, and the desired number of output camera settings. Figure 8 818).

[0139] At block 1506, process 1500 includes determining, from the plurality of data points, a data point corresponding to a camera setting having an image quality metric value that is closest to the target image quality metric value. In some examples, process 1500 includes removing, from the plurality of data points, one or more data points having the same metric value for the selected image quality metric (e.g., as described above with reference to Figure 8 804).

[0140] In some examples, process 1500 includes receiving an indication (e.g., from a Figure 6 In this example, the selected image quality metric is associated with a particular camera setting. In some cases, process 1500 includes removing from the plurality of data points one or more data points corresponding to one or more camera settings that have a lower score than the particular camera setting (e.g., as described above with reference to Figure 8 In some examples, process 1500 includes removing from the plurality of data points one or more data points corresponding to one or more camera settings that have the same metric value for the selected image quality metric and have a lower score than the particular camera setting (e.g., as described above with reference to FIG. Figure 8 804).

[0141] In some examples, process 1500 includes determining a direction for adjustment of the image quality metric based on the indication of selecting the image quality metric, including a decrease in the image quality metric. Process 1500 may include removing from the plurality of data points one or more data points corresponding to one or more camera settings that have a greater metric value for the selected image quality metric than a particular camera setting (e.g., as described above with reference to FIG. 1 ). Figure 8 In some examples, removing one or more data points from the plurality of data points results in a set of data points. In some aspects, process 1500 includes sorting the set of data points in descending order (e.g., as described above with reference to Figure 8 810).

[0142] In some examples, process 1500 includes determining a direction for adjustment of the image quality metric based on the indication of selecting the image quality metric, including an increase in the image quality metric. Process 1500 may include removing from the plurality of data points one or more data points corresponding to one or more camera settings that have a smaller metric value for the selected image quality metric than a particular camera setting (e.g., as described above with reference to FIG. 1 ). Figure 8 As described above, removing one or more data points from the plurality of data points results in a set of data points. In some examples, process 1500 includes sorting the set of data points in ascending order (e.g., as described above with reference to Figure 8 814).

[0143] In some examples, process 1500 includes outputting information associated with the determined data point for display. In some examples, process 1500 includes debugging an image signal processor (ISP) using camera settings corresponding to the determined data point.

[0144] In some examples, the processes described herein (e.g., process 700, process 800, process 1200, process 1500, and / or other processes described herein) can be performed by a computing device or apparatus. In one example, process 700, process 800, process 1200, and / or process 1500 can be performed by device 101 or Fig.16 In some cases, the device 101 performs the following operations in addition to Figure 1 In addition to the components shown in Fig.16Components of a computing device 1600. The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a connected watch or smart watch, or other wearable device), a server computer, an autonomous vehicle, a robotic device, and / or any other computing device having resource capabilities to perform the processes described herein, including process 700, process 800, process 1200, and / or process 1500. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination of the above, and / or other components. The network interface may be configured to transmit and / or receive data based on the Internet Protocol (IP) or other types of data.

[0145] Components of a computing device may be implemented in circuits. For example, a component may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0146] Process 700, process 800, process 1200 and process 1500 are shown as logical flow charts, and their operations represent a series of operations that can be implemented with hardware, computer instructions or a combination of the above. In the case of computer instructions, the operations represent computer executable instructions stored on one or more computer-readable storage media, and when these instructions are executed by one or more processors, the operations are performed. Generally, computer executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be interpreted as a limitation, and any number of the operations described can be combined in any order and / or in parallel to implement each process.

[0147] In addition, the processes described herein may be performed by hardware: under the control of one or more computer systems configured with executable instructions, implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, or a combination of the above. As described above, the code may be stored, for example, in the form of a computer program including multiple instructions executable by one or more processors on a computer-readable or machine-readable storage medium. The computer-readable or machine-readable storage medium may be non-transitory.

[0148] Fig.16 An example computing device architecture 1600 of an example computing device that can implement the various techniques described herein is shown. For example, computing device architecture 1600 can be part of device 101 (including camera 105) and can be used to implement any of the processes described herein (including process 700, process 800, process 1200, and / or process 1500). The components of computing device architecture 1600 are shown as being in electrical communication with each other using connections 1605 such as a bus. Example computing device architecture 1600 includes a processing unit (CPU or processor) 1610 and computing device connections 1605. Computing device connections 1605 couple various computing device components including computing device memory 1615, such as read-only memory (ROM) 1620 and random access memory (RAM) 1625, to processor 1610.

[0149] The computing device architecture 1600 may include a cache of high-speed memory directly connected to the processor 1610, in close proximity to the processor 1610, or integrated as part of the processor 1610. The computing device architecture 1600 may copy data from the memory 1615 and / or the storage device 1630 to the data cache 1612 for quick access by the processor 1610. In this way, the cache can achieve performance improvements and avoid delays when the processor 1610 waits for data. These and other modules can control or be configured to control the processor 1610 to perform various actions. Other computing device memories 1615 are also available. The memory 1615 may include a variety of different types of memory with different performance characteristics. The processor 1610 may include any general-purpose processor and hardware or software services, such as service 1 (1632), service 2 (1634), and service 3 (1636) stored in the storage device 1630, configured to control the processor 1610 and a dedicated processor in which software instructions are integrated into the processor design. Processor 1610 may be a self-contained system including multiple cores or processors, a bus, a memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0150] To enable user interaction with the computing device architecture 1600, the input device 1645 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, and the like. The output device 1635 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, and the like. In some cases, a multi-modal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 1600. The communication interface 1640 can generally govern and manage user input and computing device output. There is no limitation on the operation of any particular hardware arrangement, so the basic features herein can be easily substituted for improved hardware or firmware arrangements, just as it is enhanced.

[0151] Storage device 1630 is a non-volatile memory and can be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic tape cartridge, a flash memory card, a solid-state storage device, a digital versatile disk, a cassette, a random access memory (RAM) 1625, a read-only memory (ROM) 1620, and a mixture of the above. Storage device 1630 may include services 1632, services 1634, and services 1636 for controlling processor 1610. Other hardware or software modules are contemplated. Storage device 1630 may be connected to computing device connection 1605. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer-readable medium that is connected to the necessary hardware components, such as processor 1610, connection 1605, output device 1635, etc., to perform the function.

[0152] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media and / or data capable of storing, containing or carrying instructions. Computer-readable media may include non-temporary media in which data may be stored and which do not include carrier waves and / or transient electronic signals that are propagated wirelessly or via a wired connection. Examples of non-temporary media may include, but are not limited to, disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or storage devices. Computer-readable media may store codes and / or machine-executable instructions thereon, which may represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0153] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media expressly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0154] Specific details are provided in the above description to provide a comprehensive understanding of the embodiments and examples provided herein. However, it should be understood by those of ordinary skill in the art that these embodiments can be practiced without these specific details. For the sake of clarity of explanation, in some instances, the present technology can be presented as including independent functional blocks, which include functional blocks including steps or routines in methods including devices, device components, software or a combination of hardware and software. Additional components other than the components shown in the figure and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form, so as not to confuse the embodiments in terms of unnecessary details. In other instances, in order to avoid making the embodiments obscure, known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details.

[0155] The above various embodiments can be described as processes or methods, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams or block diagrams. Although the flow chart can describe the operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. The process is terminated when its operation is completed, but may have other steps not included in the figure. The process may correspond to a method, function, process, subroutine, subprogram, etc. When the process corresponds to a function, its termination may correspond to the function returning to the calling function or main function.

[0156] The processes and methods according to the above examples can be implemented using available computer executable instructions stored or from a computer readable medium. Such instructions may include, for example, instructions and data that cause or configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. Parts of the computer resources used may be easily accessible via a network. Computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media may be used to store instructions, information used, and / or information created during the methods according to the above examples. Computer readable media include disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0157] The equipment implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may adopt any of a variety of form factors. When implemented in the form of software, firmware, middleware, or microcode, the program code or code segments that perform the necessary tasks (e.g., computer program products) may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or plug-in cards. As another example, such functions may also be implemented on circuit boards between different chips or between different processes performed in a single device.

[0158] Instructions, media for transmitting such instructions, computing resources for executing the same, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0159] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application. However, it should be understood by those skilled in the art that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application are described in detail herein, it should be understood that the inventive concept can be embodied and adopted in many aspects, and the appended claims are intended to be interpreted as including these variations, unless limited to the prior art. The various features and aspects of the above-mentioned application can be used individually or in combination. Moreover, without departing from the broader spirit and scope of this specification, the embodiment can be used in any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that in alternative embodiments, these methods can be performed in an order different from the described order.

[0160] It should be understood by those of ordinary skill that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0161] Where a component is described as being “configured to” perform some operation, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor, or other suitable electronic circuit) to perform the operation, or any combination thereof.

[0162] The phrase "coupled to" refers to any component that is physically connected to another component, directly or indirectly, and / or any component that communicates directly and / or indirectly with another component (e.g., connected to another component via a wired or wireless connection, and / or other suitable communication interface).

[0163] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the group or multiple members of the group (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0164] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above in terms of their functions. Whether this function is implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. Technicians can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing deviations from the scope of the application.

[0165] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. These techniques may be implemented in any of a variety of devices. Various devices include, for example, general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses (including applications in wireless communication device mobile phones and other devices). Any features described as modules or components may be implemented together in an integrated logic device or separately implemented in discrete interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including program code. The program code includes instructions for executing one or more of the above methods when executed. A computer-readable data storage medium may form a part of a computer program product, which may include grouped materials. Computer-readable media may include memory or data storage media, for example, random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. Additionally or alternatively, these techniques may be implemented at least in part by a computer-readable communication medium. The computer-readable communication medium carries or transmits program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as a propagated signal or wave.

[0166] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0167] Illustrative examples of the present disclosure include:

[0168] Aspect 1: A method for determining one or more camera settings, the method comprising: receiving an indication of selecting an image quality metric for adjustment; determining a target image quality metric value for the selected image quality metric; and determining, from a plurality of data points, a data point corresponding to a camera setting having an image quality metric value closest to the target image quality metric value.

[0169] Aspect 2: The method according to aspect 1, wherein the indication of selecting the image quality metric includes a direction of adjustment.

[0170] Aspect 3: The method according to aspect 2, wherein the direction of adjustment includes a decrease in the image quality metric or an increase in the image quality metric.

[0171] Aspect 4: The method according to any one of Aspects 1 to 3, further comprising: removing one or more data points having the same metric value for the selected image quality metric from the plurality of data points.

[0172] Aspect 5: The method according to any one of aspects 1 to 4, further comprising: receiving an indication of selecting a specific camera setting for adjustment, wherein the selected image quality metric is associated with the selected specific camera setting.

[0173] Aspect 6: The method according to aspect 5, further comprising: removing one or more data points corresponding to one or more camera settings having a lower score than the selected specific camera setting from the plurality of data points.

[0174] Aspect 7: The method according to Aspect 5 further includes: removing one or more data points corresponding to one or more camera settings having the same metric value for the selected image quality metric and having a lower score than the selected specific camera setting from the plurality of data points.

[0175] Aspect 8: The method according to any one of Aspects 1 to 7 further includes: determining a direction of adjustment for the image quality metric based on an indication of selecting the image quality metric, including a reduction in the image quality metric; and removing one or more data points from a plurality of data points corresponding to one or more camera settings having a greater metric value for the selected image quality metric than the selected specific camera setting.

[0176] Aspect 9: The method according to aspect 8, wherein removing one or more data points from the plurality of data points generates a group of data points. The method further comprises: sorting the group of data points in descending order.

[0177] Aspect 10: The method according to any one of Aspects 1 to 9 further includes: determining a direction of adjustment for the image quality metric based on an indication of selecting the image quality metric, including an increase in the image quality metric; and removing one or more data points from a plurality of data points corresponding to one or more camera settings having smaller metric values ​​for the selected image quality metric than the selected specific camera setting.

[0178] Aspect 11: The method according to aspect 10, wherein removing one or more data points from the plurality of data points generates a group of data points. The method further comprises: sorting the group of data points in ascending order.

[0179] Aspect 12: The method according to any one of Aspects 1 to 11 further includes: determining a metric factor based on a metric value of the selected image quality metric, data points among multiple data points having extreme values ​​of the selected image quality metric, and the number of multiple data points; and determining a target image quality metric value for the selected image quality metric based on the metric factor and the metric value of the selected image quality metric.

[0180] Aspect 13: The method according to Aspect 12 also includes: receiving an indication of selecting an adjustment strength for an image quality metric; and determining a target image quality metric value for the selected image quality metric based on a metric factor, a metric value of the selected image quality metric, and the adjustment strength for the image quality metric.

[0181] Aspect 14: The method according to Aspect 12 also includes: receiving an indication of selecting a desired number of output camera settings; and determining a target image quality metric value for the selected image quality metric based on a metric factor, a metric value of the selected image quality metric, and the desired number of output camera settings.

[0182] Aspect 15: The method according to Aspect 12 also includes: receiving an indication of selecting an adjustment strength for an image quality metric; receiving an indication of selecting a desired number of output camera settings; and determining a target image quality metric value for the selected image quality metric based on a metric factor, a metric value of the selected image quality metric, an adjustment strength for the image quality metric, and the desired number of output camera settings.

[0183] Aspect 16: The method according to any one of aspects 1 to 15, further comprising: outputting information associated with the determined data points for display.

[0184] Aspect 17: The method according to any one of aspects 1 to 16 further comprises: debugging the image signal process using the camera settings corresponding to the determined data points.

[0185] Aspect 18: The method according to any one of aspects 1 to 17, wherein selecting the image quality metric for adjustment is based on a selection of a graphical element of a graphical user interface.

[0186] Aspect 19: The method according to aspect 18, wherein the graphical element includes an option to increase the image quality metric or decrease the image quality metric.

[0187] Clause 20: The method of clause 18, wherein the graphical element is associated with a displayed image having an adjustment value for the image quality metric.

[0188] Aspect 21: The method according to any one of aspects 1 to 20, wherein selecting the image quality metric for adjustment is based on selecting a displayed image frame having an adjustment value for the image quality metric.

[0189] Clause 22: The method of any one of clauses 1 to 21, wherein the camera settings are associated with one or more image signal processor settings.

[0190] Aspect 23: An apparatus for determining one or more camera settings, the apparatus comprising: a memory (e.g., implemented in a circuit) and a processor coupled to the memory. The processor is configured to: receive an indication of selecting an image quality metric for adjustment; determine a target image quality metric value for the selected image quality metric; and determine, from a plurality of data points, a data point corresponding to a camera setting having an image quality metric value closest to the target image quality metric value.

[0191] Clause 24: The apparatus of Clause 23, wherein the indication of selecting the image quality metric comprises a direction of adjustment.

[0192] Aspect 25: The apparatus according to aspect 24, wherein the direction of adjustment includes a decrease in the image quality metric or an increase in the image quality metric.

[0193] Aspect 26: An apparatus according to any one of aspects 23 to 25, wherein the processor is configured to: remove one or more data points having the same metric value for the selected image quality metric from the plurality of data points.

[0194] Aspect 27: An apparatus according to any one of aspects 23 to 26, wherein the processor is configured to: receive an indication of selecting a specific camera setting for adjustment, wherein the selected image quality metric is associated with the selected specific camera setting.

[0195] Clause 28: The apparatus of Clause 27, wherein the processor is configured to: remove from the plurality of data points one or more data points corresponding to one or more camera settings having a lower score than the selected specific camera setting.

[0196] Aspect 29: An apparatus according to Aspect 27, wherein the processor is configured to: remove from the plurality of data points one or more data points corresponding to one or more camera settings having the same metric value for the selected image quality metric and having a lower score than the selected specific camera setting.

[0197] Aspect 30: An apparatus according to any one of Aspects 23 to 29, wherein the processor is configured to: based on an indication of selecting an image quality metric, determine that the direction of adjustment for the image quality metric includes a reduction in the image quality metric; and remove from a plurality of data points one or more data points corresponding to one or more camera settings having a greater metric value for the selected image quality metric than the selected specific camera setting.

[0198] Aspect 31: The apparatus according to Aspect 30, wherein the processor is configured to: sort the set of data points in descending order.

[0199] Aspect 32: An apparatus according to any one of Aspects 23 to 31, wherein the processor is configured to: based on an indication of selecting an image quality metric, determine that a direction of adjustment for the image quality metric includes an increase in the image quality metric; and remove from a plurality of data points one or more data points corresponding to one or more camera settings having a smaller metric value for the selected image quality metric than the selected specific camera setting.

[0200] Aspect 33: The apparatus according to Aspect 32, wherein the processor is configured to: sort the set of data points in ascending order.

[0201] Aspect 34: An apparatus according to any one of Aspects 23 to 33, wherein the processor is configured to: determine a metric factor based on a metric value of a selected image quality metric, data points among a plurality of data points having extreme values ​​of the selected image quality metric, and the number of the plurality of data points; and determine a target image quality metric value for the selected image quality metric based on the metric factor and the metric value of the selected image quality metric.

[0202] Aspect 35: An apparatus according to Aspect 34, wherein the processor is configured to: receive an indication of selecting an adjustment strength for an image quality metric; and determine a target image quality metric value for the selected image quality metric based on a metric factor, a metric value of the selected image quality metric, and the adjustment strength for the image quality metric.

[0203] Aspect 36: An apparatus according to Aspect 34, wherein the processor is configured to: receive an indication of selecting a desired number of output camera settings; and determine a target image quality metric value for the selected image quality metric based on a metric factor, a metric value of the selected image quality metric, and the desired number of output camera settings.

[0204] Aspect 37: An apparatus according to Aspect 34, wherein the processor is configured to: receive an indication of selecting an adjustment strength for an image quality metric; receive an indication of selecting a desired number of output camera settings; and determine a target image quality metric value for the selected image quality metric based on a metric factor, a metric value of the selected image quality metric, an adjustment strength for the image quality metric, and the desired number of output camera settings.

[0205] Aspect 38: An apparatus according to any one of aspects 23 to 37, wherein the processor is configured to: output information associated with the determined data point for display.

[0206] Clause 39: The apparatus according to any one of Clauses 23 to 38, wherein the processor is configured to: debug the image signal process using the camera settings corresponding to the determined data points.

[0207] Clause 40: The apparatus according to any one of Clauses 23 to 39, wherein the selection of the image quality metric for adjustment is based on a selection of a graphical element of a graphical user interface.

[0208] Aspect 41: The apparatus according to aspect 40, wherein the graphical element includes an option to increase the image quality metric or to decrease the image quality metric.

[0209] Aspect 42: The apparatus according to aspect 40, wherein the graphical element is associated with a displayed image having an adjustment value for the image quality metric

[0210] Clause 43: The apparatus of any one of clauses 23 to 42, wherein selecting the image quality metric for adjustment is based on selecting a displayed image frame having an adjustment value for the image quality metric.

[0211] Clause 44: An apparatus according to any one of Clauses 23 to 43, wherein the camera settings are associated with one or more image signal processor settings.

[0212] Clause 45: The apparatus according to any one of Clauses 23 to 44, further comprising a display configured to display one or more image frames.

[0213] Clause 46: The apparatus according to any one of Clauses 23 to 45, further comprising a camera configured to capture one or more image frames.

[0214] Aspect 47: An apparatus according to any one of aspects 23 to 46, wherein the apparatus is a mobile device.

[0215] Aspect 48: A computer-readable storage medium storing instructions, which, when executed, cause one or more processors to perform any one of the operations of Aspects 1 to 22.

[0216] Aspect 49: An apparatus comprising means for performing any one of the operations of Aspects 1 to 22.

Claims

1. A method for determining one or more camera settings, the method comprising: include: receiving, via the camera commissioning graphical user interface, an indication of a first selection of a particular camera setting; receiving, via the camera commissioning graphical user interface, an indication of a second selection of an image quality metric for adjustment, wherein the selected image quality metric represents a quality of one or more image frames based on the selected particular camera setting; receiving, via the camera commissioning graphical user interface, an indication of a third selection of an adjustment to the selected image quality metric; determining a target image quality metric value for the selected image quality metric based on the selected adjustment to the image quality metric; as well as determining, from a plurality of data points, a data point corresponding to a camera setting having an image quality metric value closest to the target image quality metric value, wherein the second selection of the image quality metric for adjustment is based on a selection of a graphical element of the graphical user interface, and the graphical element includes an option to increase the image quality metric or to decrease the image quality metric, and Wherein, updated camera settings are generated based on the first selection, the second selection, and the third selection, and the updated camera settings are displayed on the camera debugging graphical user interface.

2. The method according to claim 1, in, The indication of the second selection of the image quality metric includes a direction of adjustment.

3. The method according to claim 2, in, The direction of the adjustment includes a decrease in the image quality metric or an increase in the image quality metric.

4. The method according to any one of claims 1 to 3, further comprising: include: One or more data points having the same metric value for the selected image quality metric are removed from the plurality of data points.

5. The method according to any one of claims 1 to 3, in, The selected image quality metric is associated with the specific camera settings selected.

6. The method according to claim 5, further comprising: include: One or more data points corresponding to one or more camera settings having a lower score than the selected particular camera setting are removed from the plurality of data points.

7. The method according to claim 5, further comprising: include: One or more data points corresponding to one or more camera settings having the same metric value for the selected image quality metric and having a lower score than the selected specific camera setting are removed from the plurality of data points.

8. The method according to any one of claims 1 to 3 or 6 to 7, further comprising: include: determining, based on the indication of the second selection of the image quality metric, that a direction of adjustment for the image quality metric includes a decrease in the image quality metric; as well as One or more data points corresponding to one or more camera settings having greater metric values ​​for the selected image quality metric than the selected particular camera setting are removed from the plurality of data points.

9. The method according to claim 8, in, Removing the one or more data points from the plurality of data points produces a set of data points, the method further comprising: Sort the set of data points in descending order.

10. The method according to any one of claims 1 to 3, 6 to 7 or 9, further comprising: include: determining, based on the indication of the second selection of the image quality metric, that a direction of adjustment for the image quality metric comprises an increase in the image quality metric; as well as One or more data points corresponding to one or more camera settings having smaller metric values ​​for the selected image quality metric than the selected particular camera setting are removed from the plurality of data points.

11. The method according to claim 10, in, Removing the one or more data points from the plurality of data points produces a set of data points, the method further comprising: Sort the set of data points in ascending order.

12. The method according to any one of claims 1-3, 6-7, 9 or 11, further comprising: include: determining a metric factor based on a metric value of the selected image quality metric, a data point from the plurality of data points having an extreme value of the selected image quality metric, and a number of the plurality of data points; as well as The target image quality metric value for the selected image quality metric is determined based on the metric value of the selected image quality metric and the metric factor.

13. The method according to claim 12, further comprising: include: receiving an indication of selecting an adjustment level to be made to an image quality metric; as well as The target image quality metric value for the selected image quality metric is determined based on the metric value of the selected image quality metric, the metric factor, and the adjustment strength to the image quality metric.

14. The method according to claim 12, further comprising: include: receiving an indication of selecting a desired number of output camera settings; as well as The target image quality metric value for the selected image quality metric is determined based on the metric value of the selected image quality metric, the metric factor, and the desired number of output camera settings.

15. The method according to claim 12, further comprising: include: receiving an indication of selecting an adjustment level to be made to an image quality metric; receiving an indication of selecting a desired number of output camera settings; as well as The target image quality metric value for the selected image quality metric is determined based on the metric value of the selected image quality metric, the metric factor, the adjustment strength to the image quality metric, and the desired number of output camera settings.

16. The method according to any one of claims 1-3, 6-7, 9, 11 or 13-15, further comprising: include: Information associated with the determined data points is output for display.

17. The method according to any one of claims 1-3, 6-7, 9, 11 or 13-15, further comprising: include: The image signal process is debugged using the camera settings corresponding to the determined data points.

18. The method according to any one of claims 1-3, 6-7, 9, 11 or 13-15, in, The camera settings are associated with one or more image signal processor settings.

19. An apparatus for determining one or more camera settings, include: a memory configured to store one or more camera settings; as well as a processor coupled to the memory and configured to: receiving, via the camera commissioning graphical user interface, an indication of a first selection of a particular camera setting; receiving, via the camera commissioning graphical user interface, an indication of a second selection of an image quality metric for adjustment, wherein the selected image quality metric represents a quality of one or more image frames based on the selected particular camera setting; receiving, via the camera commissioning graphical user interface, an indication of a third selection of an adjustment to the selected image quality metric; determining a target image quality metric value for the selected image quality metric based on the selected adjustment to the image quality metric; as well as determining, from a plurality of data points, a data point corresponding to a camera setting having an image quality metric value closest to the target image quality metric value, wherein the second selection of the image quality metric for adjustment is based on a selection of a graphical element of the graphical user interface, and the graphical element includes an option to increase the image quality metric or to decrease the image quality metric, and Wherein, updated camera settings are generated based on the first selection, the second selection, and the third selection, and the updated camera settings are displayed on the camera debugging graphical user interface.

20. The device according to claim 19, in, The indication of the second selection of the image quality metric includes a direction of adjustment, the direction of adjustment including a decrease in the image quality metric or an increase in the image quality metric.

21. The device according to any one of claims 19 or 20, in, The selected image quality metric is associated with the specific camera settings selected.

22. The device according to claim 21, in, The processor is configured to: One or more data points corresponding to one or more camera settings having a lower score than the selected particular camera setting are removed from the plurality of data points.

23. The device according to claim 21, in, The processor is configured to: One or more data points corresponding to one or more camera settings having the same metric value for the selected image quality metric and having a lower score than the selected specific camera setting are removed from the plurality of data points.

24. The device according to any one of claims 19-20 or 22-23, in, The processor is configured to: determining, based on the indication of the second selection of the image quality metric, that a direction of adjustment for the image quality metric includes a decrease in the image quality metric; as well as One or more data points corresponding to one or more camera settings having greater metric values ​​for the selected image quality metric than the selected particular camera setting are removed from the plurality of data points.

25. The apparatus of any of claims 19-20 or 22-23, further comprising at least one of a display configured to display one or more image frames and a camera configured to capture one or more image frames.

26. A non-transitory computer readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: receiving, via the camera commissioning graphical user interface, an indication of a first selection of a particular camera setting; receiving, via the camera commissioning graphical user interface, an indication of a second selection of an image quality metric for adjustment, wherein the selected image quality metric represents a quality of one or more image frames based on the selected particular camera setting; receiving, via the camera commissioning graphical user interface, an indication of a third selection of an adjustment to the selected image quality metric; determining a target image quality metric value for the selected image quality metric based on the selected adjustment to the image quality metric; as well as determining, from a plurality of data points, a data point corresponding to a camera setting having an image quality metric value closest to the target image quality metric value, wherein the second selection of the image quality metric for adjustment is based on a selection of a graphical element of the graphical user interface, and the graphical element includes an option to increase the image quality metric or to decrease the image quality metric, and Wherein, updated camera settings are generated based on the first selection, the second selection, and the third selection, and the updated camera settings are displayed on the camera debugging graphical user interface.

Images