Image processing device, image sensor, image processing method, and program

Abstract

To control filter processing to be applied in the vicinity of a boundary between areas different in exposure condition, in accordance with a degree of contribution of the exposure condition to a level difference of grayscale that is generated in the boundary.SOLUTION: A filter information setting unit calculates an average image value (area boundary average value) of a predetermined pixel group including pixels in contact with an area boundary, from an image including multiple areas different in exposure condition for imaging, and calculates an absolute value of an area boundary average difference which is a difference between the area boundary average values of adjacent areas and represents a level of difference of grayscale which is generated on the area boundary. The filter information setting unit generates a histogram for each of the exposure conditions, for the absolute value of the area boundary average difference in order to determine a degree of contribution of the exposure conditions to the level difference of grayscale generated in the area boundary. The filter information setting unit executes filter processing on pixels located near a boundary with an area group of which the number of area boundaries where an absolute value of the area boundary average difference is smaller than a predetermined average difference threshold exceeds a predetermined number-of-boundary threshold, for each of the exposure conditions, based on the histogram.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing technique for an image that includes a plurality of regions that are photographed under different exposure conditions. [Background technology]

[0002] It is generally known that the dynamic range of image sensors such as CCDs and CMOSs ​​used in imaging devices such as digital cameras and digital video cameras is narrower than the dynamic range of the natural world. Therefore, when a scene with a wide dynamic range (high dynamic range) is captured using conventional methods, problems such as crushed shadows and blown-out highlights occur.

[0003] In Patent Document 1, an imaging surface is divided into multiple regions, and an imaging sensor is provided that can control exposure conditions such as exposure time and readout gain for each region. By controlling the exposure conditions for each region based on information from previously captured images, it is possible to capture high dynamic range scenes. Images captured under different exposure conditions for each region have different brightness levels. Therefore, based on the exposure conditions for each captured region, pixel values ​​are corrected by applying a gain to generate a single image with uniform brightness and reduced gradation gaps.

[0004] Patent Document 2 proposes a filter processing technique for smoothly connecting the boundaries of areas with different exposure conditions when generating a high dynamic range image by combining images captured with different exposure amounts. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-136205 [Patent Document 2] Japanese Patent Application Laid-Open No. 2014-220760 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when the imaging surface is divided into multiple fine regions, the region boundaries may coincide with the edges of objects, and in such cases, gradation steps occur at the region boundaries. In such cases, where the contribution of the object edges is greater than the exposure conditions, it is desirable to leave the gradation steps as they are without applying filtering to suppress the gradation steps.

[0007] However, with the conventional technology, there is a problem in that it is difficult to control the filtering process applied near the boundary between areas with different exposure conditions in accordance with the contribution of the exposure conditions to the step in gradation that occurs at the boundary. [Means for solving the problem]

[0008] In order to solve the above problem, an image processing device that performs correction for each area based on the exposure conditions of each area for an image including a plurality of areas that are different in exposure conditions at the time of image capture, performs filtering on pixels located near the boundary between adjacent areas in the plurality of areas. A filtering means for performing Based on the contribution of the exposure conditions to the step in the gradation occurring at the boundary, At least one of enabling, disabling, and filter strength of the filtering process is selected for each pixel. Control The aforementioned The present invention is characterized by including a filter processing means. [Effects of the Invention]

[0009] The present invention makes it possible to control the filtering process applied near the boundary between regions with different exposure conditions in accordance with the contribution of the exposure conditions to the step in gradation that occurs at the boundary. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image processing apparatus according to a first embodiment. [Figure 2] Conceptual diagram showing area division of the image sensor [Figure 3] 1 is a flowchart showing a process for setting filter information according to the first embodiment. [Figure 4] Figure showing the region boundary average values [Figure 5] Figure showing an example of the absolute value of the region boundary mean difference [Figure 6] A diagram showing an example of a histogram of the absolute value of the region boundary mean difference [Figure 7] FIG. 10 is a diagram showing an example of a histogram of the absolute value of the area boundary average difference under each exposure condition; [Figure 8] FIG. 10 is a diagram showing the relationship between the histogram of the area boundary mean difference for each exposure condition and the mean difference threshold value. [Figure 9] An example of applying filtering to the boundary of a region [Figure 10] 1 is a flowchart showing a process for setting filter information according to the first embodiment. [Figure 11] FIG. 1 is a diagram showing an example of exposure conditions for each region in a frame. [Figure 12] FIG. 10 is a diagram showing an example of the number of regions with the same exposure conditions in a frame. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of an image processing apparatus according to a second embodiment. [Figure 14] 10 is a flowchart showing a process for setting filter information according to the second embodiment. [Figure 15] FIG. 10 is a diagram showing an example of an image output when generating level difference information in the second embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of the configuration of an image processing apparatus according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION [Example]

[0011] A first embodiment of the present invention will be described with reference to FIG.

[0012] 1 is a block diagram showing an image processing apparatus according to this embodiment, in which 101 denotes the image processing apparatus according to this embodiment.

[0013] The image sensor 102 is composed of a color filter and an image sensor such as a CMOS. Figure 2 shows an image sensor 1021 divided into regions of the image sensor 102. The image sensor 1021 is composed of a plurality of regions 1022, and each region 1022 is composed of a plurality of pixels 1023.

[0014] 1 receives exposure conditions set individually for each region from the sensor control unit 107 (described later), and is able to capture images under different exposure conditions for each region. The exposure conditions are defined by the exposure time and readout gain.

[0015] The area-specific correction unit 103 generates high dynamic range image data by correcting the captured image data obtained by capturing an image using the image sensor 102 under exposure conditions individually set for each area, based on the exposure conditions set for each area.

[0016] The filter information setting unit 104 sets parameters for the filter processing unit 105 (described later) based on the pixel values ​​of the high dynamic range image data output from the region-by-region correction unit 103 and the exposure conditions for each region output from the information storage unit 106. In other words, the filter processing unit 105 is controlled by the parameters set in the filter information.

[0017] The filter processing unit 105 receives filter processing parameters from the filter information setting unit 104 and performs filter processing on pixels near the boundary between areas with different exposure conditions at the time of image capture. The operations of the filter information setting unit 104 and the filter processing unit 105 will be described in detail later.

[0018] The information storage unit 106 stores the exposure conditions set for each region. There are no particular limitations on how the exposure conditions stored in the information storage unit 106 are calculated. For example, although not shown, the image processing device 101 may calculate and determine a luminance value for each region from pixel data obtained by capturing an image with the image sensor 102, or the exposure conditions may be set by an external controller (not shown).

[0019] The sensor control unit 107 receives the exposure conditions set for each region from the information storage unit 106, and controls the image sensor 102 to capture an image by individually setting the exposure conditions for each region.

[0020] Next, the operations of filter information setting section 104 and filter processing section 105 will be described in detail with reference to the flowchart of FIG.

[0021] In S301, the filter information setting unit 104 calculates the average pixel value (area boundary average value) of a predetermined pixel group including pixels adjacent to the area boundary from the pixel values ​​of the high dynamic range image data output from the area correction unit 103.

[0022] In S302, the filter information setting unit 104 calculates the absolute value of the region boundary mean difference, which is the difference between the region boundary mean value of one of two adjacent regions and the region boundary mean value of the other. The absolute value of the region boundary mean difference represents the magnitude of the difference in pixel values ​​between pixels on either side of the region boundary, and thus represents the magnitude of the gradation step that occurs at the region boundary.

[0023] The calculation of the region boundary average value and the absolute value of the region boundary average difference will be described using FIG. 4. In FIG. 4, 401, 402, 403, 404, and 405 indicate unit regions for which exposure conditions can be individually set. Note that an example will be described in which five regions, 401 to 405, are captured under four exposure conditions, A, B, C, and D. The filter information setting unit 104 calculates the average pixel value of a group of pixels in a rectangular region at the top, bottom, left, and right ends of each region. This rectangular region consists of at least pixels that contact the region boundary, and may be any number of pixels from the region boundary. In region 401, the average pixel value at the top end is a, the average pixel value at the bottom end is c, the average pixel value at the left end is f, and the average pixel value at the right end is g.

[0024] Next, the filter information setting unit 104 calculates the absolute value of the region boundary average difference. Note that the absolute value of the region boundary average difference is calculated only between regions with different exposure conditions. Here, the explanation will be given using region 401 as an example. The filter information setting unit 104 calculates the absolute value of the difference between the average pixel value b at the top end of region 401 and the average pixel value a at the bottom end of region 402. Similarly, calculations are made for the left and right ends of region 401. That is, the absolute value of the difference between the average pixel value f of the rectangular region at the left end of region 401 and the average pixel value e of the rectangular region at the right end of region 405 is calculated. The absolute value of the difference between the average pixel value g of the rectangular region at the right end of region 401 and the average pixel value h of the rectangular region at the left end of region 403 is calculated. For the rectangular region at the bottom of region 401, the absolute value of the region boundary average difference is not calculated because the exposure conditions of region 404, which is adjacent to and below region 401, are the same as those of region 401.

[0025] As described above, when the exposure conditions of adjacent regions differ at the top, left, right, and bottom edges of one region of interest, the absolute value of the region boundary average difference is calculated.

[0026] Next, in S303, the filter information setting unit 104 generates a histogram for each exposure condition for the absolute value of the calculated area boundary average value difference in order to determine the degree of contribution of the exposure condition to the gradation step that occurs at the area boundary. The generation of the histogram will be explained using FIGS. 5 and 6.

[0027] FIG. 5 shows an example of the absolute value of the area boundary average difference under exposure condition A. In FIG. 5, 501 indicates a unit area for which exposure conditions can be individually set, and an example will be described here in which there are a total of 36 areas, each of which is 6 x 6. Of the 36 areas, there are 9 areas that were captured under exposure condition A. If an adjacent area of ​​the 9 areas was captured under an exposure condition different from exposure condition A, the absolute value of the area boundary average difference is calculated (the area indicated by the thick line in FIG. 5). A histogram of the absolute values ​​of the area boundary average difference under exposure condition A is generated based on the calculated values.

[0028] FIG. 6 shows a histogram of the absolute values ​​of the area boundary average differences for the area group under exposure condition A in the example of FIG. 5. The histogram's classes (horizontal axis) represent the absolute values ​​of the area boundary average differences, and the frequency (vertical axis) represents the number of area boundaries (hereinafter referred to as the area boundary number). 601 represents the number of area boundaries whose absolute values ​​of the area boundary average differences are 0 to 999, and 602 represents the number of area boundaries whose absolute values ​​of the area boundary average differences are 1000 to 1999. The same applies to 603, 604, 605, and 606. Note that the width of each interval of the histogram classes is set to 1000 here, but this is not a limitation. For example, the intervals may be in increments of 100, or the width may be set to an arbitrary value as a parameter. In FIG. 6, there are 15 areas whose absolute values ​​of the area boundary average differences are 0 to 999, so 601 is 15.

[0029] The processing explained above in Figures 5 and 6 is performed for each exposure condition. Figure 7 shows an example of a histogram created for a group of regions for each exposure condition. Here, the example shows a case where six exposure conditions, A, B, C, D, E, and F, are used.

[0030] Next, in S304, the filter information setting unit 104 determines, based on the generated histogram of the absolute values ​​of the area boundary average differences for each exposure condition, whether the number of area boundaries for which the absolute value of the area boundary average difference is less than a predetermined average difference threshold exceeds a predetermined boundary number threshold. This is the degree of contribution of the exposure condition to the gradation step that occurs at the area boundary. Specifically, if the number of area boundaries exceeds the boundary number threshold, the process proceeds to S305, where filter processing is enabled; otherwise, the process proceeds to S306, where filter processing is disabled. The boundary number threshold will be described later.

[0031] The determination in S304 will be explained below with reference to Fig. 8. The filter information setting unit 104 sets a predetermined threshold (hereinafter referred to as the mean difference threshold) for the histogram of the absolute values ​​of the region boundary mean differences, and counts the number of region boundaries whose absolute values ​​of the region boundary mean differences are smaller than the mean difference threshold. If the absolute value of the region boundary mean difference is larger than the predetermined mean difference threshold, it is assumed that the region boundary mean difference is not due to a deviation in linearity caused by differences in exposure conditions, but is due to the edge of the object, and is not counted.

[0032] This average difference threshold is set in advance in the filter information setting unit 104 before capturing an image using the image processing device of this embodiment. There are no particular limitations on the method for determining the average difference threshold, and for example, the average difference threshold may be determined by machine learning such as deep learning on a PC using an image captured by the image sensor 102 of this embodiment.

[0033] In the image processing device of the present invention, the pixel values ​​of the pixel groups in each region are corrected based on the exposure conditions in the region-specific correction unit 103, and the bit width is expanded during this process. Therefore, in this description, it is assumed that the average difference threshold is set to 3000.

[0034] For example, Figure 8 shows the number of area boundaries for which the absolute value of the area boundary average difference is less than the average difference threshold for each exposure condition. Under exposure condition A, the number of area boundaries for which the absolute value of the area boundary average difference is less than the average difference threshold is 20. Under exposure condition B, the number of area boundaries for which the absolute value of the area boundary average difference is less than the average difference threshold is 3. Similarly, for all exposure conditions, the number of area boundaries for which the absolute value of the area boundary average difference is less than the average difference threshold is counted.

[0035] In the present invention, the number of region boundaries whose absolute value of the region boundary mean difference is less than the mean difference threshold is counted, but this is not limited to this. For example, a lower limit may be set for the absolute value of the region boundary mean difference to be counted, such as counting the number of region boundaries whose absolute value of the region boundary mean difference is 1000 or more and less than 3000. Alternatively, a histogram with a finer class width may be generated, and the number of region boundaries whose absolute value of the region boundary mean difference is 500 or more and less than 2000 may be counted.

[0036] Next, the filter information setting unit 104 determines whether the number of region boundaries whose absolute values ​​of region boundary average differences are less than a predetermined average difference threshold exceeds a boundary count threshold. The boundary count threshold is a threshold for the number of region boundaries whose absolute values ​​of region boundary average differences are less than the average difference threshold. The boundary count threshold is set in advance in the filter information setting unit 104. The method for determining the boundary count threshold is not particularly limited, and it may be determined, for example, from the total number of region boundaries in the frame. In the case of 6 × 6 = 36 regions, the total number of region boundaries in the frame is (36 × 4 - 6 × 4) / 2 = 60. In this example, the boundary count threshold is set to "12" (one-fifth of the total number of region boundaries, "60"). In this case, among the exposure conditions shown in FIG. 8, only exposure condition A has the number of region boundaries whose absolute values ​​of region boundary average differences are less than the average difference threshold exceeding 12. Therefore, in S304, it is determined that the number of region boundaries whose absolute values ​​of region boundary average differences are less than the predetermined average difference threshold exceeds the predetermined boundary count threshold (Yes) only under exposure condition A.

[0037] In S305, the filter information setting unit 104 sets the filter information so as to enable filtering for all area boundaries of areas under exposure conditions where the absolute value of the area boundary average difference is less than a predetermined average difference threshold and the number of area boundaries exceeds a predetermined boundary number threshold.

[0038] If there are many region boundaries whose absolute values ​​of the region boundary mean differences are less than the mean difference threshold, this means that there are many region boundaries where the exposure conditions are estimated to contribute more to the region boundary mean difference than the object edges. In this embodiment, if the number of region boundaries for each exposure condition exceeds one-fifth of the total number of region boundaries in the frame, it is determined that there are many region boundaries where the contribution is estimated to be other than the object edges. Furthermore, because the absolute values ​​of the region boundary mean differences are calculated for region boundaries with different exposure conditions, it is possible to know the exposure conditions that cause steps at region boundaries with different exposure conditions.

[0039] In S306, the filter information setting unit 104 sets the filter information so as to disable the filter processing on the region boundary. Note that S306 can be omitted if the filter processing is disabled in advance for all exposure conditions and the filter processing is performed only when the filter processing is enabled.

[0040] In S307, the filter information setting unit 104 outputs the filter information set in S305 and S306 to the filtering processing unit 105.

[0041] By performing the processing shown in the flowchart of FIG. 3, the filter information setting unit 104 can set filter information based on the exposure conditions.

[0042] In S305 and S306, whether or not to enable filter processing on region boundaries is set uniformly for each region, but whether or not to enable filter processing may be set for each region boundary. In this case, more information needs to be set for the filter processing unit 105, but more detailed control is possible. For example, information may be set to enable filter processing on region boundaries between regions under exposure condition A and regions under other exposure conditions, and to disable filter processing on region boundaries between regions under exposure condition A.

[0043] Next, the processing of the filter processing unit 105 will be described. Based on the filter information output from the filter information setting unit 104, the filter processing is performed on the pixel values ​​of pixels located near the boundary of the region for which the filter processing is set. The type of filter is not particularly limited. For example, a general filter such as an epsilon filter or a Gaussian filter may be used.

[0044] FIG. 9 shows pixels on which the filter processing unit 105 performs filter processing. In FIG. 9, 901 is a unit area for which exposure conditions can be set individually. 902 indicates a group of pixels located on an area boundary. In the figure, R indicates red pixels, G1 and G2 indicate green pixels, and B indicates blue pixels. 903 and 904 indicate area boundaries. The filter processing unit 105 performs filter processing on pixel values ​​located near the area boundary. Here, an example will be described in which a filter is applied within a range of four pixels from the area boundary, as pixels located near the area boundary. However, the range of pixels to which the filter is applied is not limited to this, and any range can be set.

[0045] 9, pixel group 905 surrounded by a thick line is a pixel within four pixels from the region boundary, and filter processing is applied to it. On the other hand, pixel group 906 other than pixel group 905 is more than four pixels away from the region boundary, and therefore filter processing is not applied to it. As set by filter information setting unit 104, if filter processing is enabled, filter processing unit 105 performs filter processing on the vicinity of a predetermined region boundary, and if filter processing is not enabled, filter processing is not performed.

[0046] Although the above description has been given of filtering pixels within a predetermined number of pixels from the boundary of a region and calculating pixel values ​​after filtering, the present invention is not limited to this. For example, a weighting coefficient may be set according to coordinates within the region, and after calculating pixel values ​​after filtering, a weighted average may be calculated using the weighting coefficient from the pixel values ​​before filtering and the pixel values ​​after filtering, and this weighted average value may be used as the final pixel value.

[0047] Note that, although the filter information that the filter information setting unit 104 sets in the filter processing unit 105 is explained as an example of information on whether or not to enable filter processing, it is not limited to this and may further include, for example, information for controlling the filter strength.

[0048] In this embodiment, the filter strength is determined according to the distribution of absolute values ​​of area boundary average differences for exposure conditions where the number of areas with the same exposure condition in a frame exceeds a predetermined number. The method for setting filter information in this case will be described using the flowchart in Figure 10. In Figure 10, the processes of S301, S302, S303, and S304 are the same as those in Figure 3, so their description will be omitted.

[0049] In S1004, the filter information setting unit 104 receives the exposure conditions for each region from the information storage unit 106 and counts the number of regions in the frame that have the same exposure conditions. FIG. 11 shows the exposure conditions for each 6×6 region. FIG. 12 shows the number of regions for each exposure condition in the case of FIG. 11. As shown in FIG. 12, in FIG. 11, the number of regions captured under exposure condition A is 9. The number of regions captured under exposure condition B is 5. The number of regions captured under exposure condition C is 10. The number of regions captured under exposure condition D is 5. The number of regions captured under exposure condition E is 3. The number of regions captured under exposure condition F is 4.

[0050] In S1005, the filter information setting unit 104 determines whether there is an exposure condition in which the number of regions exceeds a predetermined region number threshold. If the number of regions exceeds the predetermined region number threshold (Yes), the process proceeds to S1006, and if the number of regions is equal to or less than the predetermined region number threshold (No), the process proceeds to S1007. Here, the region number threshold is a threshold for the number of regions with the same exposure condition in a frame, and is set in advance in the filter information setting unit 104.

[0051] In S1006, the filter information setting unit 104 sets a filter strength for the filter information based on the exposure condition that exceeds the region number threshold. The details of the filter strength setting will be described later.

[0052] In S1007, the filter information setting unit 104 sets the filter information to disable the filtering process.

[0053] The filter strength setting in S1006 will now be described with reference to FIGS. 8 and 12. The filter information setting unit 104 applies a region count threshold to the number of regions in a frame that have the same exposure condition. Here, the region count threshold will be described as "6." The region count threshold may be set in advance from outside the image processing device 101, or may be determined by the filter information setting unit 104 in accordance with the total number of regions in the frame. There are no particular limitations on the method for determining the region count threshold, but here, the value "6" is specified, which is the number of regions in the frame "36 (=6×6)" divided by the number of types of exposure condition (A, B, C, D, E, F), "6."

[0054] As shown in Fig. 12, the exposure conditions in Fig. 11 that exceed the region count threshold are exposure conditions A and C. Therefore, the filter information setting unit 104 sets the filter strength based on the histograms of exposure conditions A and C. Here, the filter strength uses the filter coefficient of a Gaussian filter. A Gaussian filter is a commonly known filter used for smoothing, etc., and smooths by weighting neighboring pixel values ​​according to the distance from the pixel of interest.

[0055] In the histograms for exposure conditions A and C shown in Fig. 8, the filter coefficient is set according to the largest number of area boundaries whose absolute value of the area boundary average difference is less than the average difference threshold (3000). Under exposure condition A, the area boundaries with absolute values ​​of the area boundary average difference between 0 and 999 are most frequently distributed, so the filter coefficient is set according to this range of values.

[0056] The above method makes it possible to control the filter strength according to the distribution of absolute values ​​of region boundary average differences for exposure conditions where the number of regions with the same exposure condition in a frame exceeds a predetermined number. Note that in S1006 and S1007, the filter strength for each region boundary is set to a uniform value. However, different filter strengths may be set for each region boundary. In this case, the amount of filter information output to the filter processing unit 105 increases, but more precise control is possible. For example, for each region boundary between a region under exposure condition A and a region under another exposure condition, filter processing information is set according to the absolute value of the region boundary average difference for exposure condition A corresponding to each region boundary. Similarly, for each region boundary between a region under exposure condition C and a region under another exposure condition, filter information is set according to the absolute value of the region boundary average difference for exposure condition C corresponding to each region boundary. Filter information may be set to disable filter processing for region boundaries between regions under the same exposure condition.

[0057] In this embodiment, 101 is described as an image processing device, but is not limited to this. 101 may be configured such that image sensor 102 is a single semiconductor chip, and the remaining components 103, 104, 105, 106, and 107 are separate semiconductor chips, with the two semiconductor chips bonded together to form a stacked image sensor. [Example]

[0058] Second Embodiment A second embodiment of the present invention will be described with reference to Fig. 13. In Fig. 13, reference numeral 1301 denotes a block diagram showing an image processing device in this embodiment. In Fig. 1, the image sensor 102, the area-specific correction unit 103, the filter processing unit 105, and the sensor control unit 107 are the same as those in the first embodiment, and therefore descriptions thereof will be omitted.

[0059] Reference numeral 1304 denotes a filter information setting unit that receives exposure conditions and step information for each region from an information holding unit 1306 (to be described later) and sets filter information in the filter processing unit 105 .

[0060] An information storage unit 1306 stores exposure conditions for each region and level difference information transmitted from a level difference information generation unit 1308 (to be described later).

[0061] Reference numeral 1308 denotes a level difference information generation unit that receives pixel data from the filtering unit 105 and generates level difference information for each region. In this embodiment, when the level difference information generation unit 1308 generates level difference information, the filtering process of the filtering unit 105 is disabled.

[0062] The processing of the filter information setting unit 1304, information holding unit 1306, and level difference information generating unit 1308 will be described below with reference to the flowchart in Fig. 14. The processing of this embodiment is roughly divided into two processing steps. In the first processing step, filter processing is disabled and level difference information is generated, and in the second processing step, filter processing is performed by adjusting the filter information based on the level difference information.

[0063] Steps S1401 to S1403 in FIG. 14 are processes for generating level difference information, and are performed at a stage prior to normal imaging, such as during shipping tests of the image processing device 1300, for example.

[0064] First, the first process will be described.

[0065] In S1401, the sensor control unit 107 receives the exposure conditions for each region from the information storage unit 1306 and controls the image sensor 102 to capture an image. At this time, all exposure conditions are used comprehensively, and an image of a flat object, for example, a white board, is captured and output from the image sensor 102 to the region-specific correction unit 103.

[0066] In S1402, the area-specific correction unit 103 receives the exposure conditions for each area from the information storage unit 1306, and performs correction for each area according to the exposure conditions.

[0067] Fig. 15 shows an image output when generating level difference information. In Fig. 15, 1501 indicates a flat object such as a white board, and 1502 indicates the exposure conditions for each region (region-specific exposure conditions). Here, the explanation will be given assuming that there are six regions (3 x 2 regions) and six exposure conditions, A, B, C, D, E, and F. Also, exposure condition A is the condition under which the brightest image can be captured, followed by conditions B, C, D, and E under which the image can be captured progressively darker, with exposure condition F being the exposure condition under which the darkest image can be captured.

[0068] Reference numeral 1503 denotes an image output from the image sensor 102 when each region is imaged under the region-specific exposure conditions 1502. The output image 1503 indicates that the pixel value increases as the image approaches white, and decreases as the image approaches black.

[0069] Reference numeral 1504 denotes an image in which correction has been performed for each region based on the exposure condition by the region-by-region correction unit 103. Here, it is assumed that only the region captured under exposure condition E has pixel values ​​that differ from those of regions captured under other exposure conditions. From this image 1504, it can be seen that a relatively large difference in pixel values ​​occurs at the boundary between the region under exposure condition E corrected by the region-by-region correction unit 103 and the regions under other exposure conditions. Therefore, in such a case, it is desirable to adjust the filter strength for pixels near the boundary with the region under exposure condition E to be higher.

[0070] In S1403, the filter processing unit 105 receives the corrected image data from the area correction unit 103, and outputs the corrected image data to the level difference information generation unit 1308 without filtering.

[0071] In S1404, the level difference information generation unit 1308 receives the unfiltered and corrected image data output from the filter processing unit 105, and generates the average pixel value of the pixel group of each region as level difference information. Note that here, the level difference information is the average pixel value of the pixel group of each region, but the average pixel value of the pixel group of all regions may be used as a reference pixel value, and the difference between the reference pixel value and the average pixel value of the pixel group of each region may be used as level difference information. The generated level difference information is output to the information storage unit 1306.

[0072] By the above method, step information for each exposure condition is generated.

[0073] Next, the second process will be described.

[0074] Steps S1405 to S1407 are normal imaging processes for imaging by setting exposure conditions suitable for the imaging scene for each region, and performing filter processing by adjusting filter information based on level difference information that has been generated and stored in advance.

[0075] In S1405, the sensor control unit 107 receives the exposure conditions for each region from the information storage unit 1306, controls the imaging sensor 102 to capture an image of the object, and outputs the captured image to the region-by-region correction unit 103. The region-by-region correction unit 103 corrects the captured image based on the exposure conditions for each region stored in the information storage unit 1306, and outputs the corrected image to the filter processing unit 105.

[0076] In S1406, the filter information setting unit 1304 adjusts the filter information based on the exposure conditions of each region and the level difference information of the corresponding exposure conditions received from the information storage unit 1306, and outputs the filter information to the filter processing unit 105. The filter information setting unit 1304 calculates the difference between the pixel values ​​as level difference information of the exposure conditions of the region to be filtered and the pixel values ​​as level difference information of the exposure conditions of the adjacent region, and determines the relative filter strength between the exposure conditions based on the magnitude of the calculated difference in pixel values. There are no particular limitations on how the filter strength is determined, and for example, the filter coefficients of a Gaussian filter may be set based on the absolute value of the difference.

[0077] Note that the filter strength set here is used to determine the relative filter strength between exposure conditions, and is therefore used to adjust the filter strength set in the subsequent filter processing in S1407 using the filter strength set in S1406.

[0078] In S1407, the filtering unit 105 performs filtering based on the filter information set by the filter information setting unit 1304. This filtering is the same as that described with reference to FIGS. 3 and 10 in the first embodiment, and therefore will not be described here.

[0079] By using the above method, step information is generated in advance, and filter information is adjusted based on the step information linked to the exposure conditions, making it possible to perform appropriate filter processing on the boundaries of areas with different exposure conditions in accordance with the correction characteristics for each exposure condition of the area-specific correction unit 103.

[0080] In this embodiment, the image processing device 1301 has been described as an image processing device, but the present invention is not limited to this. The image sensor 102 constituting the image processing device 1301 may be configured as a single semiconductor chip, and the remaining components 103, 1304, 105, 1306, and 107 may be configured as separate semiconductor chips, and the two semiconductor chips may be bonded together to form a stacked image sensor. [Example]

[0081] A third embodiment of the present invention will be described with reference to Fig. 16. Fig. 16 is a block diagram showing an image processing apparatus according to this embodiment. In Fig. 16, reference numeral 1601 denotes the image processing apparatus according to this embodiment.

[0082] The image sensor 102, the area correction unit 103, the filter processing unit 105, and the sensor control unit 107 are the same as those in the first embodiment, and therefore a description thereof will be omitted.

[0083] The following describes the processing performed by the filter information setting unit 1604 and the information holding unit 1606 in this embodiment. The filter information setting unit 1604 performs the processing shown in Figures 3 and 10, and adjusts the filter information set by this processing based on temperature information about the image sensor 102, as will be described later.

[0084] The information storage unit 1606 receives and stores temperature information from the image sensor 102 using a method not shown. The temperature information can be obtained by placing a temperature sensor near the image sensor and acquiring information from the temperature sensor, or it can be estimated from the dark current component of the image sensor 102. The image sensor is energized even when not receiving light, and the current that flows there is called dark current. When pixel values ​​are read from the image sensor 102 in this state where no light is received, the pixel values ​​indicated are at a level corresponding to the magnitude of the dark current. The pixel value component resulting from this dark current is called the dark current component. Generally, image sensors are provided with a light-shielding area to estimate the dark current component. Because the value of the dark current component changes depending on the temperature of the image sensor, it is possible to estimate the temperature from the dark current component.

[0085] The filter information setting unit 1604 adjusts the filter information based on the temperature information of the image sensor 102 and the exposure conditions of each region from the information storage unit 1606. That is, the filter information setting unit 1604 adjusts the information related to the filter strength based on the temperature information of the image sensor 102 acquired from the information storage unit 1606. The filter strength here is used to determine the relative filter strength according to the temperature difference, similar to that described in the second embodiment.

[0086] Although the temperature information used in the description is related to a single temperature of the image sensor 102, this is not limiting. For example, temperature information related to multiple temperatures for each region of the image sensor 102 may be acquired, and the filter information may be adjusted based on the multiple temperatures for each region. In this case, even if the filter processing shown in FIGS. 3 and 10 is set to be disabled, the filter information setting unit 1604 adjusts the filter information so that the filter processing is enabled for region boundaries where the temperature difference between regions exceeds a predetermined value. Furthermore, when the filter processing shown in FIGS. 3 and 10 is enabled, the filter information setting unit 1604 may adjust the filter information so that the filter strength is increased for region boundaries where the temperature difference between regions exceeds a predetermined value.

[0087] By adjusting the filter information based on the temperature information using the above method, it becomes possible to perform appropriate filtering on the region boundary.

[0088] In this embodiment, the image processing device 1601 has been described as an image processing device, but the present invention is not limited to this. The image sensor 102 constituting the image processing device 1601 may be configured as a single semiconductor chip, and the remaining components 103, 1604, 105, 1606, and 107 may be configured as separate semiconductor chips, and the two semiconductor chips may be bonded together to form a stacked image sensor.

[0089] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0090] 101 Image processing device 102 Image sensor 103 Region-specific correction section 104 Filter information setting section 105 Filter processing section 106 Information holding unit 107 Sensor control unit

Claims

1. An image processing device that performs area-by-area correction on an image including multiple areas that are captured under different exposure conditions, based on the exposure conditions of each area, a filtering means for performing filtering on pixels located near a boundary between adjacent regions in the plurality of regions, the filtering means controlling at least one of enabling and disabling the filtering and the filter strength for each pixel based on the contribution of the exposure conditions to a step in gradation occurring at the boundary; An image processing device comprising:

2. the filtering means controls the filtering process for pixels located near a boundary between the adjacent regions having different exposure conditions, based on a distribution for each exposure condition of a difference in pixel value between pixels on either side of the boundary that defines the gradation step.

2. The image processing device according to claim 1, wherein:

3. the difference in pixel values ​​between pixels on either side of the boundary is a boundary mean difference, which is the difference between an average pixel value of a predetermined pixel group including pixels that contact the boundary of one region and an average pixel value of a predetermined pixel group including pixels that contact the boundary of the other region; the filtering means performs the filtering process on pixels located near the boundary with at least one region group having a boundary where the absolute value of the boundary mean difference is smaller than a predetermined mean difference threshold, among the region groups classified by exposure condition in the plurality of regions.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. the filtering means performs the filtering process on pixels located near the boundaries of a group of regions having boundaries where the absolute value of the boundary mean difference is smaller than a predetermined mean difference threshold, and a group of regions having a number of such boundaries greater than a predetermined boundary number threshold.

4. The image processing device according to claim 3.

5. the filtering means sets a filter strength in the filtering process based on a range of the absolute value of the boundary mean difference in which the boundaries having absolute values ​​of the boundary mean differences smaller than a predetermined mean difference threshold are most frequently distributed; 5. The image processing device according to claim 3, wherein the image processing device is a computer.

6. the filter processing means acquires level difference information indicating correction characteristics for each exposure condition in the correction for each region, and sets a filter strength for each exposure condition based on the level difference information.

6. The image processing device according to claim 1, wherein the image processing device is a computer.

7. the filtering means acquires temperature information of an image sensor that outputs an image including a plurality of regions with different exposure conditions, and controls the filtering process based on the temperature information.

7. The image processing device according to claim 1, wherein the image processing device is a computer.

8. the filtering means sets filter information relating to at least one of an enabling setting, an disabling setting, and a filter strength setting for the filtering process based on the contribution of the exposure conditions to the step in gradation occurring at the boundary, and controls the filtering process based on the filter information.

8. The image processing device according to claim 1, wherein the image processing device is a computer.

9. An image sensor comprising the image processing device according to any one of claims 1 to 8.

10. An image processing method for performing region-by-region correction on an image including a plurality of regions each having different exposure conditions at the time of image capture, based on the exposure conditions of each region, comprising: a step of performing a filtering process on pixels located near a boundary between adjacent regions in the plurality of regions, the filtering process controlling at least one of enabling, disabling, and filter strength of the filtering process for each pixel based on the contribution of the exposure conditions to a step in gradation occurring at the boundary; An image processing method comprising:

11. A program for causing a computer to function as the image processing device according to any one of claims 1 to 8.

Images