Image processing for on-chip inference

Abstract

To provide a method of performing an inference operation by an image processing circuit included in an imaging device.SOLUTION: A method of performing an inference operation includes: capturing a first image FH and a second image FL using a first value PH and a second value PL of an image capture parameter respectively; generating, for a first region of the first and second images, a first estimate EH and a second estimate EL respectively of an image quality metric that is dependent on the value of the image capture parameter; calculating a first distance DH,1...N between the first estimate and a first target level MH and a second distance DL,1...N between the second estimate and a second target level ML; and supplying a result Z1...N of the inference operation performed on the first region of either the first or second image selected based on the first and second distances.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure generally relates to the field of image sensors and image processing methods, and more particularly to devices and methods for performing inference on captured images. [Background technology]

[0002] Image-based inference involves using captured images to infer information about the environment captured in those images. For example, inference involves applying logical rules to input data, such as image data, to perform functions such as classification or regression of this data. Examples of applications of such inference techniques include object and / or event detection, including presence detection and / or motion detection, and detection or measurement of specific environmental parameters. In some cases, the rules applied by the inference algorithm are learned using machine learning techniques, and the inference algorithm is implemented, for example, by an artificial neural network. [Overview of the Initiative] [Problems that the invention aims to solve]

[0003] A challenge in implementing such an inference system is that lighting conditions can vary significantly in a given environment, for example, at different times of day and night, under various weather conditions. Therefore, without a feedback mechanism to properly adapt the sensing scheme to the scene characteristics, the quality of the captured images may degrade, potentially reducing the effectiveness of the inference algorithm. In extreme cases, inference may not be possible at all.

[0004] Therefore, there is a need for methods and devices that can perform image inference correctly under a relatively wide range of lighting conditions.

[0005] The object of the embodiments of this disclosure is to address, at least partially, one or more needs in the prior art. [Means for solving the problem]

[0006] According to one embodiment, an inference calculation method is provided, comprising: acquiring a first image using a first value of an image acquisition parameter; acquiring a second image using a second value of the image acquisition parameter that is smaller than the first value; generating a first estimate of an image quality metric for a first region of the first image by an image processing circuit; calculating a first distance between the first estimate and a first target level by an image processing circuit; generating a second estimate of an image quality metric for a first region of a second image by an image processing circuit; calculating a second distance between the second estimate and a second target level by an image processing circuit; and providing the result of an inference calculation performed on a first region of the first or second image, selected based on the first and second distances, wherein the image quality metric depends on the value of the image acquisition parameter, and the first regions of the first and second images are spatially corresponding regions.

[0007] According to one embodiment, the method further comprises: calculating a first new value of an image acquisition parameter based on at least a first estimate using an image processing circuit; acquiring a third image using the first new value of the image acquisition parameter; calculating a second new value of the image acquisition parameter based on at least a second estimate using an image processing circuit; and acquiring a fourth image using the second new value of the image acquisition parameter.

[0008] According to one embodiment, a first new value is further calculated based on a first target level, and a second new value is further calculated based on a second target level.

[0009] According to one embodiment, the method further comprises performing an inference operation on a first region of a first image to generate a first inference result, and performing an inference operation on a first region of a second image to generate a second inference result, wherein providing the result of the inference operation comprises selecting the first inference result or the second inference result based on the first and second distances.

[0010] According to one embodiment, the method further comprises comparing a first and a second distance using an image processing circuit, if the first distance is smaller than the second distance, performing an inference operation on a first region of the first image to generate a first inference result and providing the first inference result as the result of the inference operation, and if the second distance is smaller than the first distance, performing an inference operation on a first region of the second image to generate a second inference result and providing the second inference result as the result of the inference operation.

[0011] According to one embodiment, the method further comprises: generating a third estimate of the image quality metric for a second region of a first image using an image processing circuit; calculating a further first distance between the third estimate and a first target level using an image processing circuit; generating a fourth estimate of the image quality metric for a second region of the first image using an image processing circuit; calculating a further second distance between the fourth estimate and a second target level using an image processing circuit; and providing the results of an inference operation performed on a second region of the first or second image, selected based on the first and second distances, wherein the second regions of the first and second images are spatially corresponding regions.

[0012] According to one embodiment, the first new value of the image acquisition parameter is based on the minimum value of at least the first and third estimates, and the second new value of the image acquisition parameter is based on the maximum value of at least the second and fourth estimates.

[0013] According to one embodiment, the image acquisition parameter is the exposure time.

[0014] According to one embodiment, the first and second estimates of the image quality metric are the average pixel values ​​of the pixels in the first region.

[0015] According to one embodiment, the result of the inference operation indicates the confidence level of object detection in the first region.

[0016] According to one embodiment, the method further comprises using an image processing circuit to compare the result of an inference calculation with a threshold, and if the result exceeds the threshold, outputting a first and / or second image.

[0017] In a further embodiment, an imaging device is provided comprising one or more image sensors configured to acquire a first image using a first value of an image acquisition parameter and to acquire a second image using a second value of the image acquisition parameter that is smaller than the first value; and an image processing circuit configured to generate a first estimate of an image quality metric for a first region of the first image, calculate a first distance between the first estimate and a first target level, generate a second estimate of an image quality metric for a first region of the second image, calculate a second distance between the second estimate and a second target level, and provide the results of an inference operation performed on a first region of the first or second image, selected based on the first and second distances, wherein the image quality metric depends on the value of the image acquisition parameter, and the first regions of the first and second images are spatially corresponding regions.

[0018] According to one embodiment, one or more image sensors and image processing circuits are located on the same integrated circuit chip.

[0019] According to one embodiment, the image processing circuit is configured to further generate a third estimate of the image quality metric for a second region of a first image, calculate a further first distance between the third estimate and a first target level, generate a fourth estimate of the image quality metric for a second region of the first image, calculate a further second distance between the fourth estimate and a second target level, and provide the results of an inference operation performed on a second region of the first or second image, selected based on the first and second distances, wherein the second regions of the first and second images are spatially corresponding regions.

[0020] According to one embodiment, the image processing circuit is further configured to calculate a first new value for the image acquisition parameter based on the minimum value of at least the first and third estimates, acquire a third image using the first new value for the image acquisition parameter, calculate a second new value for the image acquisition parameter based on the maximum value of at least the second and fourth estimates, and acquire a fourth image using the second new value for the image acquisition parameter. [Brief explanation of the drawing]

[0021] The aforementioned features and advantages, as well as others, are described in detail below in the description of specific embodiments given as examples, and are not limited by reference to the accompanying drawings.

[0022] [Figure 1] This figure schematically illustrates an imaging device according to an exemplary embodiment of the present disclosure. [Figure 2] This figure provides a more detailed schematic representation of the imaging device shown in Figure 1, according to an exemplary embodiment. [Figure 3] This figure shows in more detail the image processing circuit of the imaging device shown in Figure 2 according to an exemplary embodiment of the present disclosure. [Figure 4] This figure shows a sequence of frames captured by the image sensor of the imaging device in Figure 2 according to an exemplary embodiment. [Figure 5]This diagram provides a more detailed schematic representation of the image processing circuit shown in Figure 2, illustrating an example where the circuit's functionality is implemented in software. [Figure 6] This flowchart shows the operation of a method for performing inference on image data according to an exemplary embodiment of this application. [Figure 7] This flowchart provides a more detailed explanation of the operation shown in Figure 6. [Modes for carrying out the invention]

[0023] Similar features are designated by similar reference numerals in various drawings. In particular, structural and / or functional features common to various embodiments may have the same reference numerals and may have identical structural, dimensional, and material properties.

[0024] Unless otherwise specified, when referring to two elements connected together, this means a direct connection without any intermediate elements other than conductors; when referring to two elements joined together, this means that these two elements can be connected, or they can be joined via one or more other elements.

[0025] In the following disclosures, when terms describing absolute positions such as “front,” “back,” “up,” “down,” “left,” and “right,” or terms describing relative positions such as “up,” “down,” “above,” and “downward,” or terms describing directions such as “horizontal” and “vertical,” unless otherwise specified, the references are to the orientation shown in the drawings or to imaging devices that are normally oriented during use.

[0026] Unless otherwise specified, the expressions “approximately,” “about,” “substantially,” and “on the order of” mean within 10%, preferably within 5%.

[0027] The term "image acquisition parameters" is used to refer to any of the parameters that can be set when an image is acquired by an image sensor, rather than a broader range of parameters. These include, for example: - A parameter that sets the exposure time, including the integration time of the photodiode or other type of photosite for each pixel and / or the shutter open time, in order to reduce the effects of data quantization and noise by setting the dynamic range based on the scene; -Parameters that set the focal plane, for example by adjusting the lens power and / or depth of field, in order to obtain a sharp image; and - A parameter for setting the gains, which includes the conversion gain of each pixel and the gain of the readout circuit at the bottom of the pixel array column, for example.

[0028] Figure 1 schematically shows an imaging device 100 according to an exemplary embodiment of the present disclosure. The device 100 comprises, for example, an image sensor 102 and an image processing circuit 104.

[0029] The image sensor 102 comprises, for example, an array of pixels, the array of which is formed on the focal plane of the image sensor 102. As is known to those skilled in the art, light from the image scene is focused onto the image sensor 102 via an optical system (not shown) which may include, for example, lenses, filters and / or other optical elements. The image sensor 102 is, for example, a visible light wavelength-sensitive CMOS sensor, but in alternative embodiments, the image sensor 102 may be of another type, including techniques that are sensitive to other light wavelengths such as infrared light.

[0030] The image processing circuit 104 is implemented, for example, by dedicated hardware. In some embodiments, the image processing circuit 104 is integrated into the same integrated circuit as the image sensor 102, but in alternative embodiments, they may be implemented on separate chips. More generally, the imaging device 100 is, for example, a fully custom CMOS system-on-chip.

[0031] The image processing circuit 104 provides the image sensor 102 with one or more image acquisition parameters (control parameters) to control the image acquisition operation. The image processing circuit 104 receives image data from the image sensor 102, for example, in the form of image frames, via an appropriate communication interface.

[0032] The image processing circuit 104 is configured, for example, to output image data in the form of a data signal. In some embodiments, one or more image processing operations are performed on the image data before outputting it. For example, these image processing operations may include filtering out noise from the raw image data provided by the image sensor 102, and / or other image processing adjustments.

[0033] Furthermore, the image processing circuit 104 is configured to perform inference based on image data in order to generate inference results, for example. For example, this inference may include applying logical rules to the image data to achieve functions such as classification or regression of the data. For example, the inference operation may include one or more of the following: - Detection of objects and / or events; - Presence detection and / or motion detection; and - Detection and / or measurement of specific environmental conditions.

[0034] In some embodiments, the image processing circuit 104 includes an artificial neural network having machine learning capabilities and trained, for example, to implement an inference algorithm. The use of artificial neural networks for performing inference on image data is well known to those skilled in the art and will not be described in detail here.

[0035] Figure 2 provides a more detailed schematic representation of the imaging device 100 shown in Figure 1, according to an exemplary embodiment.

[0036] As shown in Figure 2, the image acquisition parameters provided to the image sensor 102 by the image processing circuit 104 are, for example, a pair of parameters P Hand P L are included. Parameter P H and P L are different from each other, and the value of parameter P H is greater than the value of parameter P L .

[0037] For example, image sensor 102 provides two types of image data, the first type being captured using parameter P H in the form of, for example, image frame F H , and the second type being captured using parameter P L in the form of, for example, image frame F L . In some embodiments, a single frame F H and F L are interlaced at the output of image sensor 102, but in alternative embodiments, interlacing may be performed between a burst of two or more frames F H and a burst of two or more frames F L .

[0038] In some embodiments, image processing circuit 104 is configured to output one or all of captured frames F H and / or F L . In some cases, image processing circuit 104 is configured to output frame F H ' corresponding to frame F H after certain image processing, and frame F L ' corresponding to frame F L after certain image processing. Alternatively, raw image frames F H and / or F L are output by image processing circuit 104.

[0039] Image processing circuit 104 is also configured to output inference result Z. For example, as will be described in more detail below, inference result Z is provided for each pair of frames F H and F L that are processed together. Also, two or more consecutive frames F Hand two or more consecutive frames F L It is also possible to output result Z based on this. Each result Z is associated with frame F H and F L A single inference result about, or related to frame F H and F L Multiple regions R 1…N Set of inference results Z 1…N It is possible.

[0040] According to embodiments described herein, the inference result is selected based on distance calculation, as will be described in more detail below with reference to Figures 3 and 4, for frame F H Results based on the region, or frame F L This result is based on the corresponding region.

[0041] Figure 3 provides a more detailed schematic representation of the image processing circuit 104 of the imaging device shown in Figure 2, according to an exemplary embodiment of the present disclosure.

[0042] Figure 4 shows a sequence of frames captured by the image sensor 102 of the imaging device 100 over a certain period, during which the frame index of the frames increases, for example (frame index (time)). Figure 4 shows, in particular, seven frames F H(j-1) F Lj F Hj F L(j+1) F H(j+1) F L(j+2) , and F H(j+2) The sequence is shown, frame F Lj and F Hj The pair is the pair of frames currently being processed.

[0043] Referring to Figure 3, the image processing circuit 104 includes, for example, an automatic bracketing module 302, an inference algorithm 304, and an arbiter 306.

[0044] The automatic bracketing module 302, for example, takes frame F captured by the image sensor 102. H and FL , and each frame F H and F L Target level M for image quality metrics H and M L It receives the following. For example, as shown in Figure 3, the image sensor 102 receives alternating frames F H and F L Generates a stream of frames that include [the specified element].

[0045] In some embodiments, the captured frame F H F L Each of these comprises one or more regions corresponding to groups of pixels within the frame. In the example in Figure 4, each frame comprises N regions, with regions R1, R2 and R N This is illustrated. The region is, for example, a rectangle, but other shapes are also possible. Furthermore, in some cases, two regions do not separate exactly the same group of pixels, and / or the regions may overlap each other. In addition, although not shown in Figure 4, in some cases all the pixels in one region may be a subset of the pixels in another region. For example, one region may correspond to an entire frame, and another region may correspond to a subset of the pixels in this frame. Each corresponding region is, for example, a frame F processed together. H and F L For a given pair, they are in the same position within the frame. For example, in Figure 4, region R1 is in frame F which is processed together. Hj and F Lj Separates pixels at the same position within a frame. More generally, it separates a pair of frames F that are processed together. H and F L Region R within iThis divides pixels at the same position within these frames. The location of the region is selected, for example, so that a specific operation can be applied to a given region of the scene. For example, it may be desirable to detect objects that fill different regions in each frame depending on how close they are to the image sensor. In some cases, the region is fixed for the entire sequence of frames, as in the example in Figure 4. In an alternative embodiment, the region is in frame F H and F L It changes from one pair to the next, and the position of a particular region is adapted to track, for example, the movement of an object detected within the frame. Region R 1…N This can be the result of any combination of subsampling, binning, pooling, and / or cropping processes. For example, in some embodiments, region R 1…N This may be the result of an integer undersampling process of the captured frames, involving different shift or pooling operations such as averaging, minimum, or maximum pooling, which are applied to generate each region.

[0046] Referring again to Figure 3, the automatic bracketing module 302, for example, each frame F H Each region R 1…N Estimated level of image quality metric E H,1 ~E H,N Generates and each frame F L Each region R 1…N Estimated level E of the same image quality metric in L,1 ~E L,N It is configured to generate the image quality metric, for example, frame F H and F L The parameter P used to capture H and P L This metric depends on the selected parameter P. H and P L The values ​​are, respectively, frame F H and F L It has some influence on the level of the image quality metric. For example, parameter P H and PL When is the exposure time, the estimated image quality metric E H,1…N and E L,1…N are, for example, the average pixel level within region R 1…N , and the average is any type of average including, but not limited to, for example, an arithmetic mean, a median, a geometric mean, or an average of a tone-mapped image. Alternatively, when parameter P H and P L set a focal plane, the estimated image quality metric E H,1…N and E L,1…N are, for example, a measure of image sharpness in region R 1…N . When parameter P H and P L set a gain, the estimated image quality metric E H,1…N and E L,1…N are, for example, the pixel saturation of region R 1…N .

[0047] In some embodiments, when regions R 1…N have different regions from each other, calculation of the estimated image quality metrics E H,1…N and E L,1…N comprises resizing the regions, such that all these estimates are based on regions having the same size or resolution, for example with respect to pixels processed once.

[0048] In some embodiments, the estimated image quality metrics E H,1…N and E L,1…N are related to frame statistics representing the dynamic range of pixel values of frames F 1…N and F H and F L that are calculated independently for each region R

[0049] The estimated image quality metrics E H,1…N and E L,1…N are, for example, for parameters P H and P LThe automatic bracketing module 302 is used to adjust the parameters. The adjusted parameters are then provided to the image sensor 102 for use in subsequent image acquisition operations, for example. Referring to Figure 4, frame F Hj and F Lj Updated parameter P generated based on this H and P L For example, the subsequent frame F L(j+1) and F H(j+1) This is applied during the image acquisition process.

[0050] Estimated value of image quality metric E H,1…N and E L,1…N For example, it is also used by the automatic bracketing module 302 to estimate the image quality metric E, respectively. H,1 ~E H,N and target level M H Distance D between H,1 ~D H,N Calculate the estimated image quality metric E for each. L,1 ~E L,N and target level M L Distance D between L,1 ~D L,N The distance D is calculated. For example, in one embodiment, the distance D H,1 ~D H,N is the function dist H,i (E H,i M H ) is calculated using distance D L,1 ~D L,N is the function dist L,i (E L,i M L It is calculated using ). In some cases the distance calculation function is the same, i.e., dist H,i (..,..)=dist L,i (..,..) For example, D H,i =[abs(E H,i -M H ) and D L,i =[abs(E L,i -M L )]

[0051] The inference algorithm 304, for example, uses frame F captured by the image sensor 102. H and F L The algorithm receives the frames and performs inference on the region of these frames to generate the inference result Z. In the example in Figure 3, the inference algorithm 304 receives all frames F H and F L Inference is performed on the region, and therefore, for each frame F H Inference result Z H,1 ~Z H,N Generates and each frame F L Inference result Z L,1 ~Z L,N Generates.

[0052] Distance D generated by the automatic bracketing module 302 H,1…N and D L,1…N , and also, the inference result Z H,1…N and Z L,1…N For example, it is provided to the arbiter 306. The arbiter 306 is provided to each region R 1…N For each region, it is configured to select the inference result associated with the region having the smallest distance. In other words, for each region R from 1 to N, i is configured to select the inference result associated with the region having the smallest distance. i Regarding the inference result Z, H,i is D H,i <D L,i In this case, it is selected, and the inference result Z L,i is D L,i ≤D H,i It is selected in the following cases.

[0053] For each region, the selected inference result is, for example, the result Z from the image processing circuit 104. 1…N This forms the output set for each inference result Z. 1…NFor example, the result may be a scalar value, but depending on the inference operation, it may alternatively be a more complex result such as a vector. In some embodiments, the inference operation is a classification operation, and the inference result is, for example, the confidence level of a given label corresponding to the presence of an object in a given region R. For example, the inference algorithm is trained so that if the result is positive in a given region, this means that an object or other property has been found. Alternatively, the inference operation may be a regression operation that estimates a quantity associated with a given region R, rather than a classification operation. An example of such a regression operation may be the numbering of a particular object.

[0054] In an alternative embodiment, the inference algorithm 304 processes each frame F H All inference results regarding Z H,1 ~Z H,N , and each frame F L All inference results regarding Z L,1 ~Z L,N Instead of systematically calculating the distance D H,1…N and D L,1…N The distances may also be provided to the inference algorithm 304 by the automatic bracketing module 302, which is configured to compare distances for each region and perform inference only for the region with the smallest distance. In other words, for each region R from 1 to N, i is 1. i Regarding the inference result Z, H,i D H,i <D L,i In this case, the calculation is performed, and the inference result Z L,i D L,i ≤D H,i This is calculated in the case of [the specified condition]. The inference result selected for each region is, for example, the result Z of the image processing circuit 104, as described above. 1…N This forms the output set. Therefore, in this case, the arbiter 306 may be omitted.

[0055] Figure 5 provides a more detailed schematic representation of the image processing circuit 104, with an example where the circuit's functions are implemented by software executed by appropriate hardware. The processing circuit 104 includes, for example, a processing device 502, which comprises one or more processors under the control of instructions 504 stored in, for example, an instruction memory (RAM) 506, which is a random access memory. The processing device 502 and the RAM 506 are coupled, for example, via a bus 508. A further memory 510 is also coupled, for example, to the bus 508, and, for example, a memory section 512 stores frames captured by the image sensor 102, image acquisition parameters 514 are applied to the image sensor 102, and a target value 516 indicates the target image quality metric.

[0056] The input / output interface (I / O interface) 518 is also coupled, for example, to bus 508, enabling communication with other devices such as the image sensor 102 and other hardware of the imaging device 100.

[0057] Some or all of the functions of the image processing circuit 104 may be implemented not by software, but by one or more dedicated hardware circuits such as an ASIC (Application-Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). If the inference algorithm 304 is implemented by an artificial neural network, this artificial neural network may be implemented in software, in other words, by computational instructions and data stored in the memory of the circuit 104, or at least partially by dedicated hardware.

[0058] Figure 6 is a flowchart showing the operation of a method for performing inference on image data according to an exemplary embodiment of the present application. This method is performed, for example, by the image processing circuit 103 in Figures 2 and / or 5.

[0059] Function 601 (frame sequencer) controls the image sensor 102 by the image processing circuit 104, and sets the image acquisition parameter P H and P LInterlace frame F based on each H and F L This includes generating frame F. H This is a series of operations 602~606(F H Generated and processed by (processing), frame F L This is a series of operations 602'~606'(F L It is generated and processed by (processing).

[0060] Operation 602 (frame F) H In acquisition, frame F H This is acquired from the image sensor 102.

[0061] Similarly, operation 602' (frame F L In acquisition, frame F L This is acquired from the image sensor 102.

[0062] In operations 603-1 to 603-N (calculation), the estimated value of the image quality metric is E. H,1…N , and the inferred value Z H,1…N For example, frame F H Region R 1…N It is generated for each of them.

[0063] Similarly, in operation 603-1'~603-N' (calculation), the estimated value of the image quality metric E L,1…N , and the inferred value Z L,1…N For example, frame F L Region R 1…N It is generated for each of them.

[0064] In operations 604-1 to 604-N (calculated), the estimated image quality metric E H,1…N and target level M H Distance D between H,1…N For example, this is calculated.

[0065] Similarly, in operation 604-1'~604-N' (calculated), the estimated image quality metric E L,1…N and target level M L Distance D betweenL,1…N For example, this is calculated.

[0066] In operation 605 (calculation), frame F H Estimated value of image quality metric E H For example, the following is generated. In some embodiments, the image quality metric E H The estimated value E H,1…N It is selected as the smallest value among them.

[0067] Similarly, in operation 605', frame F L Estimated value of image quality metric E L This is generated. In some embodiments, the image quality metric E L The estimated value E L,1…N It is selected as the largest value among them.

[0068] In operation 606 (update), parameter P H For example, the estimated value E of the image quality metric. H and target level M H It will be updated based on this.

[0069] Similarly, in operation 606' (update), parameter P L For example, the estimated value E of the image quality metric. L and target level M L It will be updated based on this.

[0070] In some embodiments, updating parameters in operations 606 and 606' involves using a lookup table. Furthermore, in some embodiments, updating parameters involves P H >P L And the parameter P H and P L This includes forcing them to be different from each other. In some embodiments, M H and M L They can be the same.

[0071] In operation 607 (arbiter), region R 1…N The inference result Z1…N For example, the inference result Z H,1…N and the corresponding distance D H,1…N , and the inference result Z L,1…N and the corresponding distance D L,1…N It is generated based on this.

[0072] In some cases, the image processing circuit 104 simply processes frame F in the case of object detection or other forms of important inference results regarding one of these frames. H and / or F L It is configured to output the following. In such a case, operation 607 may output, for example, the inference result Z. 1…N Each detection threshold th d This includes comparing with Z i >th d If so, the image processing circuit 104 will determine the inference result Z L,1…N In addition, Frame F H and / or F L It was configured to output the following.

[0073] Figure 7 shows the image acquisition parameter P. H and P L Exposure time T H and T L (Here, T H >T L This is a flowchart illustrating in more detail some of the operations in Figure 6 according to an exemplary embodiment. In such cases, a relatively long exposure time T H This allows for good capture of low-light and / or low-contrast areas in the image scene, while also enabling relatively short exposure times T LThis allows for the reduction of overexposure in high-light and / or high-contrast areas. In fact, certain image scenes can have a relatively high intra-scene dynamic range, which requires a relatively high number of bits per pixel to capture correctly. Such high intra-scene dynamic ranges are particularly present in active lighting systems, such as when illuminating a scene using near-infrared LEDs (light-emitting diodes) or when an infrared filter is placed in front of the image sensor. In fact, in such cases, the intensity of reflected light from objects relatively close to the image sensor is on the order of the reciprocal of the square of the object's distance. Similar high intra-scene dynamic ranges can also occur with passive lighting, for example, when an image is captured through a window. A high dynamic range exists in image combinations by capturing frames with long and short exposure times.

[0074] Operation 701 in Figure 7 corresponds to operations 602 and 602' in Figure 6, and frame F Hj and F Lj These are the respective image acquisition parameters T Hj and T Lj It is incorporated based on this.

[0075] Operation 702 (Image G) Hj In the calculation, tone mapping and resolution reduction are used to reduce frame F Hj Image G Hj This is converted to a log2 representation, which uses, for example, the log2 representation of each pixel in an image. For example, the binary code representing the value of each pixel is converted to a representation based on the log2 transformation via the most significant bit position operator, e.g., 001XXXXX → 101(5) or 00001XXX → 011(3). For example, a pixel value encoded in 8 bits is therefore encoded in 3 bits.

[0076] Similarly, operation 702' (image G Lj (Calculation) In image G Hj Image G is generated using a similar method to tone mapping and resolution reduction. Lj This is generated.

[0077] Operation 703 (Estimated value E) H,1…N In the calculation, the estimated value E of operations 603-1 to 603-N is calculated. H,1…N For example, image G Hj It is generated based on the region R of the frame. i The estimated image quality is image G Hj Region R within i It is calculated based on the sum of the pixels, and only the most significant bit of a specific number is retained. For example, the calculation can be expressed by the following formula:

number

[0078] Similarly, operation 703' (estimated value E) L,1…N In the calculation, the estimated value E of operation 603-1'~603-N' is calculated. L,1…N For example, image G Lj It is generated based on the region R of the frame. i The estimated image quality of image G can be expressed, for example, by the following formula: Li Region R within i Calculated based on the sum of pixels:

number

[0079] Figure 7 Operation 704 (min(E) H,1…N ) and M H Based on T H,(j+1) The operation that generates the parameter 606 in Figure 6 corresponds to the operation in Figure 6 where the parameter is updated. In the example in Figure 7, the parameter T is used to capture the next frame. H,(j+1) For example, the estimated value E H,1…N It is calculated based on the smallest estimate among them. For example, parameter T H,(j+1) It is calculated based on the following formula:

number

number

number

[0080] In some embodiments, the parameter T H,(j+1) This uses a lookup table to determine the variable k H It is updated based on the following. Furthermore, in some embodiments, the estimated value E is used to speed up the convergence time. H,i If the result is a linear operation and therefore there is no tone mapping stage such as the average, then index k H The mechanism for updating index k can be based on feedback control. For example, index k H,(j+1) is equation k H,(j+1) =k H,j +[log q (M H )-log q (E H It may be updated based on ), and [.] is a function that brings the result into the integer domain, such as rounding operations or threshold functions.

[0081] Similarly, the operation 704'(max(E) in Figure 7 L,1…N ) and M L Based on T L,(j+1)The operation 606' in Figure 6, which generates the parameter, corresponds to the operation 606' in Figure 6, where the parameter is updated. In the example in Figure 7, the parameter T is used to capture the next frame. L,(j+1) For example, the estimated value E L,1…N It is calculated based on the largest estimate among them. For example, parameter T L,(j+1) It is calculated based on the following formula:

number

number

[0082] In some embodiments, the parameter T L,(j+1) This uses a lookup table to determine the variable k L It is updated based on the following. Furthermore, in some embodiments, the estimated value E is used to speed up the convergence time. L,i If the index k is the result of a linear operation such as the mean, L The mechanism for updating index k can be based on feedback control. For example, index k L,(j+1) is equation k L,(j+1) =k L,j +[log q (M L )-log q (E L It may be updated based on ), and [.] is a function that brings the result into the integer domain, such as rounding operations or threshold functions.

[0083] Target level M H and M L and estimated value E H,1…N and E L,1…NThe above example of a method for updating parameters based on is merely one example, and it will be obvious to those skilled in the art that different calculation methods can be used.

[0084] For example, in some embodiments, T H and T L The parameter is modified in steps of a fixed size ΔT, for example, equal to the minimum step size, so that the parameter is gradually modified over many cycles. For example, E H <M H If so, T H,(j+1) =T j +ΔT, or E H >M H If so, T H,(j+1) =T j -ΔT, and E L <M L If so, T L,(j+1) =T j +ΔT, or E L >M L If so, T L,(j+1) =T j -ΔT

[0085] After operations 704 and 704', the updated parameter T H,(j+1) and T L,(j+1) Further images are captured, for example, using this method. For example, in operations 705 and 705' following operations 704 and 704' respectively, j is incremented, and the method returns to operation 701 where a new frame is captured using the updated parameters. Furthermore, after operations 704 and 704' as well, the inference results are obtained for each region of the frame in operations 706-713, for example, as will be explained in more detail below. 1…N It is generated for this purpose.

[0086] Operation 706 (Inference Z H,1…N The calculation corresponds to the inference calculations of operations 603-1 to 603-N in Figure 6, and the inference result Z H,1…N Frame F Hj Region R 1…N It is generated for this purpose.

[0087] Similarly, operation 706' (inference Z L,1…N The calculation corresponds to the inference calculation of operations 603-1' to 603-N' in Figure 6, and the inference result Z L,1…N Frame F Lj Region R 1…N It is generated for this purpose.

[0088] Operation 707 (distance D H,1…N The calculation corresponds to the distance calculation of operations 604-1 to 604-N in Figure 6, and distance D H,1…N This is calculated.

[0089] Similarly, operation 707' (distance D L,1…N The calculation corresponds to the distance calculation of the operation 604-1'~604-N' in Figure 6, and distance D L,1…N This is calculated.

[0090] After operations 707 and 707', arbitration is performed in operations 708-713, which correspond to operation 607 in Figure 6. For example, in operation 708, the variable i is set to 1, and in operation 709, the distance D H,i and D L,i D is compared. H,i <D L,i In the case of (branch Y), in operation 710, the inference result Z i Frame F Hj Region R i The value Z calculated for H,i It is set to this. Otherwise (branch N), in operation 710', the inference result Z i Frame F Lj Region R i The value Z calculated for L,i It is set to . After operations 710 and 710', for example in operation 711, it is determined whether i is equal to the number N in the region. If not, in operation 712, i is incremented, and the method returns to operation 709. If i is equal to N in operation 711, the output result Z 1…N This is operation 713 (result Z 1…N Output is output via (output).

[0091] Minimum region-based estimator E HBased on the parameter T H Updated the largest region-based estimator E L Based on the parameter T L It should be noted that updating this has the advantage of essentially deriving different parameter values.

[0092] Figure 7 shows the image acquisition parameter P. H and P L An example is shown where is the exposure time, but it will be obvious to those skilled in the art how the method in Figure 7 can be adapted to other types of image acquisition parameters.

[0093] An advantage of the embodiments described herein is that, based on a relatively simple distance calculation, inference operations can be applied to the best case of two different captured images.

[0094] Furthermore, the image acquisition parameter P H and P L Exposure time T H and T L If so, the advantage is that the embodiments described herein provide a relatively low-complexity solution for performing inference operations on low-dynamic-range images, yielding performance close to that of inference performed on the same but high-dynamic-range images. Indeed, alternative solutions could be to acquire high-dynamic-range images or merge two low-dynamic-range images to generate a high-dynamic-range image, but processing such images is extremely complex. Indeed, the inference algorithm would have to be designed or trained to process such images, and therefore, given the high bit depth, the size and complexity of the inference algorithm would be very high. The inference algorithm remains relatively simple by applying the same inference algorithm to either or both of two frames acquired with different image acquisition parameters and selecting the inference result based on the regions of the two frames that best match the target image quality.

[0095] Various embodiments and modifications are described. Those skilled in the art will understand that specific features of these embodiments can be combined and that other modifications will readily arise. For example, it will be apparent to those skilled in the art: - Frame F H and F L An embodiment is described in which frames F are captured by the same image sensor, but with a certain image sensor, frame F H Capture the image and use another image sensor to capture frame F L It is also possible to incorporate it; - Frame F is processed as a pair H and F L These are images captured in relatively close time instances, but these frames do not necessarily have to be consecutive frames from the image sensor; -In some embodiments, frame F H and F L If the integration times are different, the image sensor pixels between frames F will not be reset. H and F L You may incorporate them sequentially. For example, while the pixels are being integrated, frame F L It was read in a non-destructive manner, frame F H This is incorporated after a further integration period; - Estimated value E H and E L The estimated value E H,1…N The minimum and estimated value E L,1…N Although it is described as being based on the maximum value among them, in an alternative embodiment, the estimated value E H and E L These are the estimated values ​​of the regions E, respectively. H,1…N and E L,1…N It may be calculated based on two or more of the following: - Two types of frames F are incorporated. H and F L Embodiments in which a frame exists are described, but the teachings described herein may be applied to three or more types of frames, for example, a further medium frame F MA moderate parameter P is added. M It is incorporated based on this.

[0096] Finally, the actual implementation of the embodiments and modifications described herein is within the capabilities of those skilled in the art based on the functional descriptions provided herein.

Claims

1. A method for performing inference operations, To acquire the first image using the first value of the image acquisition parameter, A second image is acquired using a second value of the image acquisition parameter that is smaller than the first value. The image processing circuit generates a first estimated value of the image quality metric for a first region of the first image. The image processing circuit calculates a first distance between the first estimated value and the first target level. The image processing circuit generates a second estimated value of the image quality metric for the first region of the second image. The image processing circuit calculates a second distance between the second estimated value and the second target level, and The image processing circuit provides the result of the inference operation performed on the first region of the first or second image, which is selected based on the first and second distances. Equipped with, The image quality metric depends on the value of the image acquisition parameter. The first region of the first and second images is a spatially corresponding region, in this method.

2. The image processing circuit calculates a first new value for the image acquisition parameter based on at least the first estimated value. To acquire a third image using the first new value of the image acquisition parameter, The image processing circuit calculates a second new value for the image acquisition parameter based on at least the second estimated value, and A fourth image is acquired using the second new value of the aforementioned image acquisition parameter. The method according to claim 1, further comprising:

3. The method according to claim 2, wherein the first new value is further calculated based on the first target level, and the second new value is further calculated based on the second target level.

4. Performing the inference operation on the first region of the first image to generate a first inference result, and Performing the inference operation on the first region of the second image to generate a second inference result. Furthermore, The method according to any one of claims 1 to 3, wherein providing the result of the inference operation comprises selecting the first inference result or the second inference result based on the first and second distances.

5. The image processing circuit compares the first and second distances. If the first distance is smaller than the second distance, the inference operation is performed on the first region of the first image to generate a first inference result, and the first inference result is provided as the result of the inference operation, and If the second distance is smaller than the first distance, the inference operation is performed on the first region of the second image to generate a second inference result, and the second inference result is provided as the result of the inference operation. The method according to any one of claims 1 to 3, further comprising the above.

6. The image processing circuit generates a third estimated value of the image quality metric for the second region of the first image. The image processing circuit calculates a further first distance between the third estimated value and the first target level. The image processing circuit generates a fourth estimated value of the image quality metric for the second region of the first image. The image processing circuit calculates a further second distance between the fourth estimated value and the second target level, and To provide the result of the inference operation performed on the second region of the first or second image, selected based on the first and second distances. Furthermore, The method according to any one of claims 1 to 3, wherein the second region of the first and second images is a spatially corresponding region.

7. The method according to claim 6, as referenced to claim 2, wherein the first new value of the image acquisition parameter is based on at least the minimum of the first and third estimates, and the second new value of the image acquisition parameter is based on at least the maximum of the second and fourth estimates.

8. The method according to any one of claims 1 to 3, wherein the image acquisition parameter is the exposure time.

9. The method according to any one of claims 1 to 3, wherein the first and second estimates of the image quality metric are the average pixel values ​​of the pixels in the first region.

10. The method according to any one of claims 1 to 3, wherein the result of the inference operation indicates the confidence level of object detection in the first region.

11. The method according to claim 10, further comprising the image processing circuit comparing the result of the inference calculation with a threshold, and outputting the first and / or second image if the result exceeds the threshold.

12. One or more image sensors configured to acquire a first image using a first value of an image acquisition parameter and to acquire a second image using a second value of the image acquisition parameter that is smaller than the first value, For the first region of the first image, a first estimate of the image quality metric is generated. A first distance is calculated between the first estimated value and the first target level. For the first region of the second image, a second estimated value of the image quality metric is generated. A second distance is calculated between the second estimated value and the second target level. The result of an inference operation performed on the first region of the first or second image, selected based on the first and second distances, is provided. An image processing circuit configured as follows: Equipped with, The image quality metric depends on the value of the image acquisition parameter. The first region of the first and second images is a spatially corresponding region in the imaging device.

13. The imaging device according to claim 12, wherein the one or more image sensors and the image processing circuit are located on the same integrated circuit chip.

14. The aforementioned image processing circuit further, For the second region of the first image, a third estimated value of the image quality metric is generated. A further first distance is calculated between the third estimated value and the first target level. For the second region of the first image, a fourth estimated value of the image quality metric is generated. A further second distance is calculated between the fourth estimated value and the second target level. The result of the inference operation performed on the second region of the first or second image, selected based on the first and second distances, is provided. It is configured in such a way, The imaging device according to claim 12 or 13, wherein the second region of the first and second images is a spatially corresponding region.

15. The aforementioned image processing circuit further, Based on at least the minimum of the first and third estimated values, a first new value of the image acquisition parameter is calculated. Using the first new value of the image acquisition parameter, a third image is acquired. Based on at least the maximum values ​​of the second and fourth estimates, a second new value for the image acquisition parameter is calculated. A fourth image is acquired using the second new value of the image acquisition parameter. The imaging device according to claim 14, configured as described above.

Images