Method and apparatus for automatic exposure

Abstract

To provide a method of controlling automatic exposure (AE).SOLUTION: An automatic exposure control method disclosed herein comprises: receiving statistics for AE control for an image from an image signal processor (ISP) coupled to an image sensor generating the image; computing an exposure value at a current time t (EV(t)) using a cost function based on target characteristics of the image, where the computation of the cost function uses the statistics; and computing AE settings for the image sensor based on EV(t).SELECTED DRAWING: Figure 2

Description

[Background technology]

[0001] Exposure is a crucial factor contributing to the quality of images captured by imaging sensor systems such as cameras. Overexposure or underexposure in areas of an image can result in loss of detail and faded colors. In real-time vision-based systems, such as advanced driver-assistance systems, industrial imaging, and security systems, captured images are provided as direct input to computer vision algorithms, and image quality affects the success of these algorithms. Many imaging sensor systems offer an automatic exposure (AE) mode that automatically calculates and adjusts exposure settings. However, current AE methods may not adequately correct the entire image. [Overview of the project]

[0002] Embodiments of the present disclosure relate to automatic exposure in an imaging sensor system. In one embodiment, a method for automatic exposure (AE) control is provided, which includes receiving statistical values ​​for AE control of an image from an image signal processor (ISP) coupled to an image sensor that generates an image; calculating an exposure value (EV(t)) at current time t using a cost function based on target characteristics of the image, wherein the cost function is calculated using statistical values; and calculating an AE setting for the image sensor based on EV(t).

[0003] In one embodiment, a system-on-a-chip (SoC) is provided, which includes an image signal processor (ISP), a memory configured to store software instructions for implementing automatic exposure (AE) control, and at least one processor coupled to the memory for executing the software instructions. The ISP is configured to generate statistics for automatic exposure (AE) control from images captured by an image sensor coupled to the ISP. The software instructions include instructions for calculating the exposure value (EV(t)) at current time t using a cost function based on target characteristics of the image. The calculation of the cost function calculates the AE setting of the image sensor based on EV(t) using the statistics and the software instructions.

[0004] In one embodiment, a method for learning the weights of a cost function for automatic exposure (AE) control includes: receiving parameters in a weight learning component coupled to a virtual imaging system; performing multiple experiments on the virtual imaging system for multiple scenes stored in a scene database coupled to the virtual imaging system, using different combinations of a candidate set of weight values ​​selected from a range of candidate weight values ​​and a starting EV selected from a set of starting EVs; and having the weight learning component calculate a quality standard for each candidate set of weight values ​​based on n EVs calculated for each experiment performed using the candidate set of weight values. The parameters include a range of candidate weight values, a set of starting exposure values ​​(EVs), and n time steps for performing the experiment. An experiment includes calculating n EVs for a given scene using a candidate set of weight values ​​and a starting EV, and the scene includes multiple images.

[0005] In one aspect, a computer system is provided, the computer system including a processor and a memory storing software instructions that, when executed by the processor, cause the calculation of quality criteria for automatic exposure (AE) control. The software instructions are for receiving parameters in a weight learning component coupled to a virtual imaging system, for performing a plurality of experiments on the virtual imaging system using different combinations of a set of candidate weight values selected from a range of candidate weight values and a start EV selected from a set of start EVs for a plurality of scenes stored in a scene database coupled to the virtual imaging system, and for calculating, by the weight learning component, a quality criterion for each set of candidate weight values based on n EVs calculated for each experiment of the plurality of experiments performed using the set of candidate weight values. The parameters include a range of candidate weight values, a set of start exposure values (EVs), and the number of time steps n for performing the experiments. An experiment includes calculating n EVs for a scene using a set of candidate weight values and a start EV, the scene including a plurality of images.

Brief Description of the Drawings

[0006] [Figure 1] It is a simplified block diagram illustrating an example of automatic exposure (AE) control in an imaging sensor system.

[0007] [Figure 2] It is a flowchart of a method for AE control in an imaging sensor system.

[0008] [Figure 3] It is a block diagram of an exemplary system for learning the values of AE weights α, β, γ.

[0009] [Figure 4] It is a flowchart of a method for learning the values of AE weights α, β, γ.

[0010] [Figure 5]This is a high-level block diagram of an example multiprocessor system-on-a-chip (SoC).

[0011] [Figure 6] This is a simplified block diagram of a computer system. [Modes for carrying out the invention]

[0012] Specific embodiments of this disclosure will be described in detail herein with reference to the accompanying figures. Similar elements in various figures are denoted by the same reference numerals for consistency.

[0013] Embodiments of this disclosure provide automatic exposure (AE) control for images in an imaging sensor system. In particular, embodiments provide a method for calculating an exposure value (EV) used to determine the AE setting of an imaging sensor. Depending on the particular imaging system, EV may be a function of exposure time (ET) (shutter speed) used to capture the image, and one or more of the following: analog gain (AG), f-number (aperture size), and digital gain (DG). For example, an imaging sensor system used in an automotive application may be configured such that the aperture size cannot be changed, and therefore EV may be a function of ET and AG. The EV calculation method described herein is based on target image characteristics that indicate an exposure setting to obtain a well-exposed image. In some embodiments, the target characteristics that form the basis of the EV calculation are the target brightness, the target ratio of low-tone pixels, and the target ratio of high-tone pixels.

[0014] Figure 1 is a simplified block diagram illustrating an example of AE control in an imaging sensor system according to several embodiments. The imaging sensor system includes a system-on-a-chip (SoC) 100 coupled to an image sensor 103 to receive raw image sensor data. The SoC 100 includes an image signal processor (ISP) 101 and a processor 102 for executing AE control software 104. The ISP 101 has the function of receiving raw image sensor data and generating a processed image by performing various image processing operations on the raw sensor data. Image processing operations may include decompanding, defective pixel correction (DPC), lens shading correction, spatial noise filtering, brightness and contrast enhancement, demosaicing, and color enhancement. The ISP 101 is also configured to generate statistical values ​​from the image for AE calculation and provide the generated AE statistics to the processor 102. The AE statistics for a given image may be a downsampled version of the image, where each pixel of the downsampled version of the image is generated by averaging the pixels of the original image.

[0015] The AE control software 104 determines the EV for one or more subsequent images captured by the image sensor 103 using AE statistics (e.g., a downsampled image) generated for the current image, referred to herein as the image at time t. The AE control software 104 calculates the exposure value (EV) using a cost function that indicates the exposure setting for obtaining a well-exposed image, based on the target characteristics of the image, i.e., the target brightness of the image and the target ratio of low-tone pixels to high-tone pixels in the image. More specifically, for each image, the cost function minimizes the normalized deviation of the average brightness of the image from the normalized deviation of the probability of low-tone pixels in the image from the normalized deviation of the target ratio of high-tone pixels, and the normalized deviation of the probability of high-tone pixels in the image from the normalized deviation of the target ratio of high-tone pixels. The formulas for these normalized deviations are shown below, and the definitions of the variables in these formulas are shown in Table 1.

[0016] The normalized deviation from the target brightness is calculated as follows: The normalized deviation from the target ratio of low-tone pixels in TIFF2023051873000002.tif414 is calculated as follows. D MB , H , L , range = |T PL - p L (t)| The normalized deviation from the target ratio of high-tone pixels is calculated as follows. D PH = |T PH - p H (t)|

Table 1

[0017] The AE control software calculates the cost function as follows. c = α -DMB × β DPL × γ -DPH Here, α, β, γ ≥ 1 are weights given as parameters to the AE control software 104. The target average brightness T MB , the target probability T of low-tone pixels PL , and the target probability T of high-tone pixels PH are also given as parameters to the AE control software 104. The AE control software 104 uses the downsampled image to calculate the brightness range B range , the average brightness I MB (t) at time t, the probability p of low-tone pixels at time t L (t), and the probability p of high-tone pixels at time t H (t). The brightness range B<00​​​​​​(t) can be calculated as the number of pixels in the image that exceed the high-tone threshold divided by the total number of pixels in the image. The low-tone threshold and high-tone threshold are provided as parameters to the AE control software 104.

[0018] In some embodiments, the values ​​of α, β, and γ may be selected based on the impact that achieving each target has on the realization of a well-exposed image. For example, if meeting the target brightness goal is a better indicator of a well-exposed image than meeting the target ratio of low-tone pixels, then α > β. In some embodiments, the values ​​of α, β, and γ are learned using machine learning techniques. Such embodiments are described below.

[0019] The AE control software 104 calculates the EV at time t as follows: TIFF2023051873000004.tif629 Here, Tc is a threshold cost value representing the minimum cost to adjust the EV. The threshold cost Tc is given as a parameter to the AE control software 104. Then, the obtained EV(t) is used to generate the exposure settings for the image sensor 103, such as AG and ET.

[0020] In some embodiments, the AE control software 104 implements a stability lock used to prevent the EV from changing based on small or irregular changes in scene brightness between images. The stability lock determines whether or not to use the calculated value of EV(t) to determine the exposure setting of the image sensor 103. Table 2 shows pseudocode for the stability lock according to some embodiments. In this pseudocode, the lock state of locked indicates whether the AE calculation is locked (not used) or not, lock count is used to prevent the EV from changing based on a single bad frame, such as an image with very high brightness compared to the previous image, Run AE indicates whether or not to use the calculated value of EV(t), and y curr is the current average scene brightness, and y avgThis is the average brightness of the previous number of images identified by the user, and T BD is, T MB This is the maximum allowable brightness deviation from [the given value]. [Table 2]

[0021] Figure 2 is a flowchart of a method for automatic exposure (AE) control in an image sensor system according to several embodiments. As an example, this method will be explained with reference to the image sensor system illustrated in Figure 1. In this method, the AE control software 104 controls the parameters α, β, γ, T C , T BD , T MB , T PL , and T PH Assume it is initialized with the value of .

[0022] First, AE statistics (e.g., downsampled image) generated for the image captured by the image sensor 103 at time t are received by the AE control software 104. The AE statistics are generated by the ISP 101 and stored in memory for access by the AE control software 103. The AE control software 104 uses the AE statistics for the image to calculate the exposure value EV(t) as described herein (202). Next, the exposure setting of the image sensor 103 is calculated based on the value of EV(t) (204) and output to the image sensor 103 (206). In some embodiments, after EV(t) is calculated, the stability lock described herein may be performed to determine whether the exposure setting of the image sensor 103 is calculated using the calculated EV(t) or whether the exposure setting is not changed.

[0023] Figure 3 is a block diagram of an exemplary system for learning the values ​​of AE weights α, β, and γ. This system includes a weight learning component 300 coupled to a virtual imaging system 302 configured to perform the above-described method for calculating EV(t), and a scene database 304 coupled to the virtual imaging system 302. The weight values ​​are learned by conducting experiments with the virtual imaging system 302 using various combinations of weight values ​​and starting EV. Based on these experimental results, quality criteria are calculated and used to determine the weight values.

[0024] The scene database 304 stores multiple scenes, each containing multiple images and corresponding EVs for those images. More specifically, for a given scene, the scene database 304 stores multiple images of that scene, each image being captured using a different EV. The EV range used to capture images can be selected to cover the EV range of the image sensor, but it does not need to include all possible EVs. In some embodiments, images corresponding to EVs within the EV range are captured sequentially in short intervals, e.g., less than one minute. Each scene in the scene database 304 may use the same EV range. For example, for a particular image sensor, the scene database 304 may contain 50 scenes with images collected at EV = {1, 1.2, 1.4, ..., 8}.

[0025] The virtual imaging system 302 includes a virtual imaging system control component 306, an image interpolation component 310, and an AE calculation component 308. The image interpolation component 310 is coupled to a scene database 304 and configured to provide an image to the AE calculation component at the specified EV. Inputs to the interpolation component 310 include a scene identifier, e.g., a scene name or number, and an EV. The image interpolation component 310 is configured to search for an image in the scene database 304 corresponding to the identified scene in order to find an image captured at the requested EV. If such an image is found, it is provided to the AE calculation component 308.

[0026] If no such image is found, the image interpolation component 310 is configured to interpolate an image corresponding to the requested EV and provide the interpolated image to the AE calculation component 308. In some embodiments, the image interpolation component 310 generates an interpolated image using two images from an identified scene captured at the EV immediately above and immediately below the requested EV. More specifically, the image interpolation component 310 interpolates image I as follows: EV(t) Generates. TIFF2023051873000006.tif653 Here, EV(t) is the requested EV, I EV(t) + This is the scene image from scene database 304 that has the closest EV greater than EV(t), and EV(t) + is, I EV(t) + It is an EV, I EV(t) - This is the scene image from scene database 304 that has the closest EV smaller than EV(t), and EV(t) - is, I EV(t) - It is an EV.

[0027] The AE calculation component 308 is configured to calculate the EV based on the image received from the image interpolation component 308. The EV is calculated as described above in this specification. TIFF2023051873000007.tif628AE Calculation component 308 calculates the brightness range B based on the image received from the image interpolation component 310. range , the average brightness I at time t MB (t), the probability of a low-gradation pixel and the probability of a high-gradation pixel at time t, p H(t) Calculate the target average brightness T. MB , target probability T of low-gradation pixels PL , target probability T of high-gradation pixels PH , threshold cost T CThe values ​​of the weights α, β, and γ are input parameters to the AE calculation component 308. In some embodiments, the AE calculation component 308 is also configured to implement the stability lock described herein.

[0028] The virtual imaging system control component 306 is configured to control the operation of the virtual imaging system 302. Inputs to the virtual imaging system control component 306 include parameters for the image interpolation component 310 and the AE calculation component 308, including the parameters described herein, as well as an initial EV, a scene identifier, and the number of time steps for running the virtual imaging system 302 for the image interpolation component 310. The virtual imaging system control component 306 includes functions to run the image interpolation component 310 and the AE calculation component 308 over the specified number of time steps. Each time step includes generating an image at the EV requested by the image interpolation component 303 (initial EV, or the EV output by the AE calculation component 308 in the previous time step) and calculating the EV based on the generated image by the AE calculation component 308. The virtual imaging system control component 306 is further configured to capture the EV output by the AE calculation component 308 after each time step and provide the captured EV to the weight learning component 300.

[0029] The weight learning component 300 is configured to cause the virtual imaging system 302 to perform experiments using candidate values ​​for α, β, and γ and various starting EVs, and to calculate a quality standard for each combination of candidate values ​​for α, β, and γ. The goal is to identify the values ​​of α, β, and γ that result in an EV within the target range of EVs identified as producing a well-exposed image in a scene in the scene database 304. The target range of EVs can be determined, for example, by having experts review all the images in the scene database 304 and identifying images with good exposures based on their opinions. The target range of EVs can then be set using the EVs of the identified images. In some embodiments, multiple experts may be used, and the target range of EVs may be determined from the opinions of multiple experts.

[0030] The weight learning component 300 receives from the user the EV goal range, the range of candidate weight values ​​for α, β, and γ to try, the set of starting EVs, and the number of time steps, as well as the T for the AE calculation component 308. MB , T PL , T PH , T C , T BD The system is configured to receive values ​​related to, as well as thresholds for low and high gradation. The weight learning component 300 instructs the virtual imaging system 302 to perform experiments over a number of time steps for each scene in the scene database 304, for each possible combination of α, β, and γ from the range of candidate weight values ​​and the starting EV, and receives a set of EVs generated by the AE calculation component 308 from the virtual imaging system 302 for each experiment. In this specification, the combination of values ​​of α, β, and γ selected from the range of candidate weight values ​​may be referred to as the candidate set of weight values. The set of EVs contains one EV value for each time step. More specifically, the weight learning component 300 receives a set of EVs for each experiment described by a 5-tuple (α, β, γ, S, EV(0)). Here, α, β, and γ are candidate weight values ​​used in the experiment, S is the scene identifier for the experiment, and EV(0) is the starting EV for the experiment.

[0031] For example, the range of candidate weight values ​​can be [1, 2, 3], the set of starting EVs can be [1, 4, 8], and the number of time steps can be 10. The possible combinations of α, β, γ, i.e., the candidate sets of weight values, are [1, 1, 1], [1, 1, 2], [1, 1, 3], [1, 2, 1], [1, 2, 2], [1, 2, 3], ... [3, 3, 3]. For each of these candidate sets of weight values, experiments are conducted for each scene using each starting EV. For example, for each scene in the database, an experiment is conducted using α, β, γ = [1, 1, 1] and a starting EV of 1. Next, for each scene in the database, an experiment is conducted using α, β, γ = [1, 1, 1] and a starting EV of 4. Next, for each scene in the database, an experiment is conducted using α, β, γ = [1, 1, 1] and a starting EV of 8. This process is repeated for all possible combinations of α, β, γ within the range of candidate weight values.

[0032] After all experiments have been conducted, the weight learning component 300 has a set of 10 EVs for each experiment. The weight learning component 300 is configured to calculate quality criteria based on the set of EVs generated by each experiment. For example, in some embodiments, the quality criteria are calculated for each possible candidate set of weight values ​​α, β, and γ, and are the ratio of experiments using candidate sets of weight values ​​that converge to the goal EV range, the probability of convergence using candidate sets of weight values, and the average convergence time using candidate sets of weight values. An experiment for a candidate set of weight values ​​converges if the EVs generated for scenes using the candidate set of weight values ​​realize EV values ​​within the goal range of EV values ​​and remain within the goal range for consecutive time steps.

[0033] For example, assuming the target range for the EV value is [2, 3], the starting EV value is 8, and the number of time steps is 10, the EVs obtained for an experiment on a certain scene are 8, 6, 4, 2, 1, 2, 2, 2, 2, and 2. In this experiment, the EV converged to the target range in time step 6 and remained converged in the last four time steps. The convergence time for a combination of α, β, and γ is the number of time steps in which the AE calculation component 308 is called until the EV is within the target range and convergence occurs in the remaining time steps of the experiment, multiplied by the frame rate of the imaging sensor used to acquire images into the scene database 304. Following the previous example, since convergence began in time step 6, the convergence time steps are 6, and the convergence time is 6 times the frame rate.

[0034] Figure 4 is a flowchart of a method for learning the values ​​of AE weights α, β, and γ according to several embodiments. As an example, this method will be explained with reference to the system illustrated in Figure 3. This method assumes that one or more experts have viewed images in the scene database 304 and that the target range of EV has been determined. First, the weight learning component 300 includes the range of candidate weight values ​​for α, β, and γ to be tried, the set of starting EVs, the number of time steps n for each experiment, and T MB , T PL , T PH, T C , T BD The virtual imaging system 302 receives parameters for learning weight values, including values ​​related to and thresholds for low and high gradation (400), and T MB , T PL , T PH , T C , T BD Initialize using values ​​related to , as well as low- and high-gradation thresholds.

[0035] The weight learning component 300 sets values ​​for α, β, and γ for the virtual imaging system 302, selected from a range of candidate weight values ​​(401), and sets a value EV(0) for the starting EV selected from a set of starting EVs (402). Next, the weight learning component 300 runs experiments on the virtual imaging system 302 for a specified number of time processes for the scenes in the scene database 304 and receives the obtained EV from the virtual imaging system 302 (406). If there is another scene 408 in the scene database for which experiments have not been conducted for the current values ​​of α, β, γ, and EV(0), the weight learning component 300 runs experiments on the virtual imaging system 302 for another scene in the scene database 304 using the current values ​​of α, β, γ, and EV(0).

[0036] When an experiment is performed using the current values ​​of α, β, γ and EV(0) for all scenes 408 in the scene database 304, if there is another starting EV 410 in the set of starting EVs that has not been used in the experiment using the current values ​​of α, β, and γ, the weight learning component 300 sets EV(0) as the next starting EV (402), performs an experiment on the virtual image processing system 302 for each scene 408 in the scene database 304 using the current values ​​of α, β, γ and EV(0), and receives the obtained EV (406).

[0037] When an experiment is performed for all scenes 408 in the scene database 304 using the current values ​​of α, β, and γ and all starting EVs 410 within the set of starting EVs, if there is another set of candidate values ​​412 for α, β, and γ within the range of candidate weight values ​​for which no experiment has been performed, the weight learning component 300 sets the values ​​of α, β, and γ to another set of values ​​within the range of weight values, and for each scene 408, it performs an experiment on the virtual imaging system 302 using the current values ​​of α, β, and γ for each starting EV 410 within the set for starting EVs (404), and receives the obtained EV (406).

[0038] When experiments are conducted for all scenes 408 in the scene database 304, for all combinations of α, β, and γ values ​​within the range of candidate weight values ​​and the value of EV(0) within the set of starting EVs, the weight learning component 300 uses the EVs from each experiment to calculate a quality criterion for each combination of α, β, and γ values, i.e., for each candidate set of weight values. The calculation of an example of a quality criterion has already been described herein. The user can then use these quality criteria to select a combination of α, β, and γ values ​​to be used for automatic exposure control in the imaging sensor system.

[0039] Figure 5 is a high-level block diagram of an exemplary multiprocessor system-on-chip (SoC) 500 that may be configured to implement an embodiment of the automatic exposure (AE) control described herein. In particular, the exemplary SoC 500 is an embodiment of the TDA4VM SoC available from Texas Instruments, Inc. A high-level description of the components of SoC 500 is provided herein. A more detailed description of the exemplary components is provided in Texas Instruments, SPRSP36J, February 2019, revised August 2021, pp. 1-323, “TDA4VM Jacinto® Automotive Processors for ADAS and Autonomous Vehicle Silicon, Revisions 1.0 and 1.1,” which is incorporated herein by reference. [Non-Patent Document 1] “TDA4VM Jacinto. Automotive Processors for ADAS and Autonomous Vehicles Silicon Revisions 1.0 and 1.1,” Texas Instruments, SPRSP36J, February, 2019, revised August, 2021, pp. 1-323

[0040] The SoC500 includes numerous subsystems across different domains, such as one dual-core 64-bit Arm® Cortex®-A72 microprocessor subsystem 504, two dual-core Arm® Cortex®-R5F MCU-based microcontroller unit (MCU) islands 506, four additional dual-core Arm® Cortex®-R5F MCUs 512 in the main domain, two C66x floating-point digital signal processors (DSPs) 508, one C71x floating-point vector DSP 510 including a deep learning matrix multiplication accelerator (MMA), and a 3D graphics processing unit (GPU) 512. The SoC500 further includes a memory subsystem 514, which includes up to 8 MB of on-chip static random access memory (SRAM), an internal DMA engine, a general-purpose memory controller (GPMC), and an external memory interface (EMIF) module (EMIF). The SoC500 also includes a capture subsystem 516 with two camera streaming interfaces, a vision processing accelerator (VPAC) 502 including one or more image signal processors (ISPs), a depth and motion processing accelerator (DMPAC) 518, and a video acceleration module 520. The SoC500 further includes a display subsystem 522, an Ethernet subsystem 524, a navigator subsystem 526, various security accelerators 528, support for system services 530, and various other interfaces 532.

[0041] The software instructions for implementing the AE control software described herein may be stored in the memory subsystem 514 (e.g., a computer-readable medium) and executed on one or more programmable processors of the SoC 500, such as the DSP 510. Furthermore, one or more ISPs in the VPAC 502 may be, for example, the embodiment of ISP 101 in Figure 1.

[0042] Figure 6 is a simplified block diagram of a computer system 600 that may be used to run the system for learning the values ​​of the AE weights α, β, and γ described herein. The computer system 600 includes a processing unit 630 having one or more input devices 604 (e.g., a mouse, keyboard, etc.) and one or more output devices such as a display 608. In some embodiments, the display 608 may be a touchscreen and therefore may also function as an input device. The display may be any suitable visual display unit, such as a computer monitor, LED, LCD, or plasma display, television, high-definition television, or a combination thereof.

[0043] The processing unit 630 includes a CPU (central processing unit) 618, memory 614, storage device 616, video adapter 612, I / O interface 610, video decoder 622, and network interface 624, all of which are connected to a bus. The bus may be one or more of several bus architectures of any kind, including a memory bus or memory controller, peripheral bus, video bus, etc.

[0044] The CPU 618 may be a suitable combination of any appropriate type of electronic data processor. For example, the CPU 618 may be one or more processors, one or more reduced instruction set computers (RISCs), one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), etc., from Intel or Advanced Micro Devices. The memory 614 may be any type of system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), flash memory, or a combination thereof. The memory 614 may also include ROM used at startup and DRAM for data storage used during program execution.

[0045] The storage device 616 (e.g., a computer-readable medium) may include any type of storage device configured to store data, programs, and other information, and to make the data, programs, and other information accessible via a bus. The storage device 616 may be one or more of the following: a hard disk drive, a magnetic disk drive, an optical disk drive, etc. Software instructions implementing the system for learning the values ​​of the AE weights α, β, and γ described herein may be stored in the storage device 616. The scene database may also be stored in the storage device 616 or accessed via the network interface 624. The software instructions may first be stored in a computer-readable medium such as a compact disk (CD), diskette, tape, file, memory, or any other computer-readable storage device, and then loaded and executed by the CPU 618. In some examples, the software instructions may also be sold as a computer program product, including the computer-readable medium and its packaging. In some cases, software instructions may be distributed to computer system 600 via removable computer-readable media (e.g., floppy disks, optical disks, flash memory, USB keys), via transmission paths from computer-readable media on another computer system (e.g., a server), or by other means.

[0046] The video adapter 612 and the I / O interface 610 provide interfaces for connecting external input / output devices to the processing unit 630. As shown in Figure 6, examples of input / output devices include a display 608 connected to the video adapter 612 and a mouse / keyboard 604 connected to the I / O interface 610.

[0047] The network interface 624 allows the processing unit 630 to communicate with a remote unit over a network. The network interface 624 may provide an interface for wired links, such as Ethernet cables, and / or wireless links, over a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, other similar types of networks, and / or any combination thereof. Other examples

[0048] While this disclosure has described a limited number of embodiments, those skilled in the art who benefit from this disclosure will understand that other embodiments can be devised that do not deviate from the scope disclosed herein.

[0049] For example, this specification describes a specific technique for interpolating images to generate an image using a desired EV. Other interpolation techniques may be used in other embodiments.

[0050] In another example, this specification describes an embodiment in which a particular cost function is used for AE control in an imaging sensor system. In other embodiments, other cost functions may be used that represent achieving an image that is considered well exposed by one or more experts.

[0051] In another example, this specification describes an embodiment in which a particular method for learning the weight values ​​of a cost function for AE control is used. In other embodiments, other learning techniques may be used.

[0052] In another example, the embodiment described herein relates to an exemplary stability lock. In other embodiments, a stability lock may not be used, or other methods may be used to implement a stability lock.

[0053] In another example, this specification describes an embodiment in which AE statistics may be downsampled images. In other embodiments, different or additional statistics may be available for AE, such as an image histogram.

[0054] Accordingly, the attached claims are intended to cover any modifications of the embodiments that fall within the true scope of this disclosure.

Claims

1. 1. A method for automatic exposure (AE) control, comprising: receiving statistics for AE control of the image sensor system; obtaining values ​​of one or more weights of a cost function, the values ​​of the one or more weights being determined using a learning process based on a plurality of images corresponding to different exposure values; determining an exposure value EV for the image sensor system using the cost function based on the received statistics; and determining AE settings for the image sensor system based on the determined exposure value EV; A method comprising:

2. 10. The method of claim 1, determining an exposure value EV for the image sensor system using the cost function is further based on target characteristics in addition to the received statistical values, the target characteristics including an average brightness of a target in the image, a target probability of low-level pixels in the image, and a target probability of high-level pixels in the image.

3. 3. The method of claim 2, wherein the cost function minimizes the normalized deviation of the average brightness of the image from the target average brightness, the normalized deviation of the probabilities of low-level pixels in the image from the target probabilities for low-level pixels, and the normalized deviation of the probabilities of high-level pixels in the image from the target probabilities for high-level pixels.

4. 4. The method of claim 3, The cost function is given by: c=a －Ｄ ＭＢ ×b Ｄ ＰＬ ×c －Ｄ ＰＨ D ＭＢ is the normalized deviation of the image mean brightness from the target mean brightness, and D ＰＬ is the normalized deviation of the low-tone pixel probabilities in the image from the target probability of the low-tone pixel, and D ＰＨ A method wherein α, β, and γ are weights in the cost function;

5. 5. The method of claim 4, the plurality of images are stored in a scene database storing a plurality of scenes, each scene corresponding to a subset of the plurality of images, each image in the subset of the plurality of images for a scene corresponding to a different exposure value, and the values ​​of α, β, and γ are determined using the learning process based further on the plurality of images as well as a goal range of exposure values.

6. 10. The method of claim 1, The method, wherein determining the exposure value EV includes determining an exposure value (EV(t)) at a current time t as a product of the cost function and an exposure value EV(t-1) at a previous time (t-1) based on determining that the cost function is greater than a minimum cost.

7. 10. The method of claim 1, The method further includes determining whether the determined exposure value EV is used to determine the AE setting based on a stability lock.

8. 1. A system on chip (SoC), comprising: an image signal processor (ISP) configured to generate statistics for auto-exposure (AE) control from images captured by an image sensor coupled to the ISP; a memory configured to store software instructions; at least one processor configured to execute the software instructions implementing auto-exposure (AE) control; Including, To implement the auto exposure (AE) control, the software instructions cause the at least one processor to: acquiring the generated statistics; obtaining values ​​for one or more weights of the cost function that are predetermined using a learning process based on a plurality of images corresponding to different exposure values; determining an exposure value EV using the cost function based on the generated statistics; determining AE settings for the image sensor based on the determined exposure value EV; A SoC includes software instructions.

9. 9. The SoC of claim 8, The SoC, wherein the exposure value EV is determined using the cost function based further on target characteristics in addition to the generated statistics, the target characteristics including a target average brightness of the image, a target probability of low-level pixels in the image, and a target probability of high-level pixels in the image.

10. 10. The SoC of claim 9, the cost function minimizes a normalized deviation of the average brightness of the image from the target average brightness, a normalized deviation of the probability of low-level pixels in the image from the target probability of low-level pixels, and a normalized deviation of the probability of high-level pixels in the image from the target probability of high-level pixels.

11. The SoC of claim 10, The cost function is given by: c=a －Ｄ ＭＢ ×b Ｄ ＰＬ ×c －Ｄ ＰＨ D ＭＢ is the normalized deviation of the image mean brightness from the target mean brightness, and D ＰＬ is the normalized deviation of the low-tone pixel probabilities in the image from the target probability of the low-tone pixel, and D ＰＨ is the normalized deviation of the probability of a high-tone pixel in the image from the target probability of the high-tone pixel, and α, β, γ are weights of the cost function.

12. The SoC of claim 11, the plurality of images are stored in a scene database that stores a plurality of scenes, each scene corresponding to a subset of the plurality of images, each image in the subset of the plurality of images for a scene corresponding to a different exposure value, and the values ​​of α, β, and γ are determined using the learning process based further on a goal range of exposure values ​​in addition to the plurality of images.

13. 9. The SoC of claim 8, The SoC, wherein determining the exposure value EV includes determining an exposure value EV(t) at a current time t as a product of the cost function and an exposure value EV(t-1) at a previous time (t-1) if the cost function is greater than a minimum cost.

14. 9. The SoC of claim 8, The SoC, wherein the software instructions further cause the at least one processor to use a stability lock to determine whether the exposure value EV is used to determine the AE setting.

15. 1. A method for learning cost function weights for auto-exposure (AE) control, comprising: receiving parameters at a weight learning component coupled to the virtual imaging system, the parameters including a range of candidate weight values, a set of starting exposure values ​​(EV), and a number n of time steps for the iterations; performing the iterative operation on the virtual imaging system using different combinations of a candidate set of weight values ​​selected from the range of candidate weight values ​​and a starting EV selected from the set of starting EVs for a plurality of scenes stored in a scene database coupled to the virtual imaging system, the scene including a plurality of images, wherein each step of the iterative operation includes calculating n EVs for a scene using the candidate set of weight values ​​and a starting EV; calculating a quality metric for each candidate set of weight values ​​based on the n EVs calculated during each step of the iteration performed by the weight learning component with the candidate set of weight values; A method comprising:

16. 16. The method of claim 15, The method, wherein the quality criteria include a proportion of iterations performed using the candidate set of weight values ​​that converge to a goal EV range, a probability of convergence using the candidate set of weight values, and an average convergence time using the candidate set of weight values.

17. 16. The method of claim 15, The method, wherein calculating n EVs for a scene further uses a cost function that includes a candidate set of weight values ​​based on a target characteristic of an image.

18. 18. The method of claim 17, The method, wherein the target characteristics include a target average brightness of the image, a target probability of low-level pixels in the image, and a target probability of high-level pixels in the image.

19. 20. The method of claim 18, the cost function minimizes the normalized deviation of the average brightness of the image from the target average brightness, the normalized deviation of the probability of low-level pixels in the image from the target probability of low-level pixels, and the normalized deviation of the probability of high-level pixels in the image from the target proportion of high-level pixels.

20. 20. The method of claim 19, The cost function is given by: c=a －Ｄ ＭＢ ×b Ｄ ＰＬ ×c －Ｄ ＰＨ D ＭＢ is the normalized deviation of the image mean brightness from the target mean brightness, and D ＰＬ is the normalized deviation of the low-tone pixel probabilities in the image from the target probability of the low-tone pixel, and D ＰＨ is a normalized deviation of the probability of high tone pixels in the image from the target proportion of high tone pixels, and α, β, γ are weights in the candidate set of weight values.

21. 16. The method of claim 15, Calculating n EVs for a scene includes: calculating a first EV of the n EVs using the candidate set of weight values, the starting EV, and a first image corresponding to the starting EV, wherein the first image is an image of the scene captured using the starting EV or an image interpolated from an image of the scene captured at a higher EV and an image of the scene captured at a lower EV; calculating a second EV of the n EVs using the candidate set of weight values, the first EV, and a second image corresponding to the first EV, wherein the second image is an image of the scene captured using the first EV or an image interpolated from an image of the scene captured at a higher EV and an image of the scene captured at a lower EV; A method comprising:

22. 1. A computer system comprising: a processor; a memory storing software instructions that, when executed by the processor, cause calculation of quality criteria for automatic exposure (AE) control; Including, The software instructions: receiving parameters including a range of candidate weight values, a set of starting exposure values ​​(EV), and a number n of time steps of the iterations at a weight learning component coupled to the virtual imaging system; performing an iterative operation on the virtual imaging system using different combinations of a candidate set of weight values ​​selected from the range of candidate weight values ​​and a starting EV selected from the set of starting EVs for a plurality of scenes stored in a scene database coupled to the virtual imaging system, the iterative operation including calculating n EVs for a scene including a plurality of images using the candidate set of weight values ​​and the starting EV; calculating, by the weight learning component, a quality metric for each candidate set of weight values ​​based on the n EVs calculated during each step of the iteration using the candidate set of weight values; a computer system including software instructions for:

23. 23. The computer system of claim 22, The quality criteria include the proportion of steps of the iterative operation performed using the candidate set of weight values ​​that converge to a goal EV range, the probability of convergence using the candidate set of weight values, and the average convergence time using the candidate set of weight values.

24. 23. The computer system of claim 22, The computer system, wherein calculating the n EVs for a scene further uses a cost function based on target characteristics of the image.

25. 25. The computer system of claim 24, The computer system, wherein the target characteristics include a target average brightness of the image, a target probability of low-level pixels in the image, and a target probability of high-level pixels in the image.

26. 26. The computer system of claim 25, the cost function minimizes a normalized deviation of the average brightness of the image from the target average brightness, a normalized deviation of the probability of low-tone pixels in the image from the target probability of low-tone pixels, and a normalized deviation of the probability of high-tone pixels in the image from the target proportion of high-tone pixels.

Images