Image processing device, image processing method, imaging device, and program

Abstract

To provide an image processing device which can correct a target pixel with high quality.SOLUTION: An imaging apparatus 100 comprises: an imaging unit 105; and an image processing unit 107. The image processing unit 107 includes a correction processing part 200 and performs pixel correction processing of an image acquired by an image pickup device of the imaging unit 105. The correction processing part 200 performs interpolation processing by extracting a small area of an input image including a target pixel in the acquired image and acquiring coordinate data of the target pixel. The correction processing part 200 generates data of the second image by performing downsampling on data of a first image in which the interpolation processing is applied to the target pixel of the extracted small area and performing upsampling on the data. The correction processing part 200 generates a composite image by combining a corresponding pixel in the second image to the target pixel of the small area.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for pixel correction processing of an image captured by an imaging device. [Background technology]

[0002] Imaging devices use CCD image sensors, CMOS image sensors, SPAD image sensors, etc. as imaging elements. CCD stands for "Charge-Coupled Device," CMOS stands for "Complementary Metal Oxide Semiconductor," and SPAD stands for "Single Photon Avalanche Diode."

[0003] For image signals acquired by image sensors, measures are required to prevent pixel defects such as scratches caused by dark current and scratches due to manufacturing factors. For example, consider the case where crosstalk occurs between pixels in a SPAD image sensor. In this case, if the deterioration caused by the scratch spreads to surrounding pixels and becomes a scratch that spans multiple pixels, the degradation of image quality may become more noticeable.

[0004] Patent Document 1 discloses an image defect correction technology for correcting image defects in radiographic images. Patent Document 2 discloses a radiographic image processing technology that can accurately correct defective pixels even when pixel defects and defects caused by scintillator scratches or dust contamination coexist in the same location. These technologies can correct not only scratches in a single pixel but also scratches that span multiple pixels. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-253668 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-187220 Summary of the Invention [Problem to be solved by the invention]

[0006] With conventional technology, correction processing for scratches that span multiple pixels can result in noticeable correction marks, making it difficult to achieve high-quality image correction. An object of the present invention is to provide an image processing device that enables higher quality correction of a pixel of interest. [Means for solving the problem]

[0007] An image processing device according to one embodiment of the present invention includes: an acquisition means for acquiring position information of a pixel of interest in an image; an extraction means for extracting an area including the pixel of interest from the image; an interpolation means for performing interpolation relating to the pixel of interest in the area extracted by the extraction means; a first processing means for downsampling an image having pixels interpolated by the interpolation means and outputting data of a first image; ,by machine learning-based calculations using the correct image and training images. The image processing system is characterized by comprising a second processing means for performing upsampling and generating data of a second image, and a synthesis means for synthesizing a pixel of interest in the region with a corresponding pixel in the second image to generate data of a third image. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device that enables higher quality correction of a pixel of interest. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an example of a basic configuration of an imaging device according to an embodiment. [Figure 2] FIG. 1 is a block diagram illustrating a configuration of a first embodiment. [Figure 3] FIG. 2 is a schematic diagram illustrating coordinate data of regions and pixels. [Figure 4]FIG. 4 is a schematic diagram illustrating pixel selection processing in the first embodiment. [Figure 5] FIG. 4 is a schematic diagram illustrating color interpolation processing in the first embodiment. [Figure 6] FIG. 3 is a schematic diagram illustrating color synthesis processing in the first embodiment. [Figure 7] FIG. 10 is a block diagram illustrating the configuration of a second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of the relationship between average luminance and the degree of image quality degradation. [Figure 9] FIG. 10 is a table showing examples of reduction ratios and enlargement ratios corresponding to degrees of image quality degradation. [Figure 10] 10A and 10B are diagrams illustrating an example of the relationship between the temperature of an imaging element and the degree of image quality degradation. [Figure 11] FIG. 10 is a block diagram illustrating the configuration of a third embodiment. [Figure 12] 10A and 10B are diagrams illustrating a reliability calculation process for pixel correction. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present invention will be described in detail below with reference to the drawings. In the embodiments, imaging devices such as digital cameras and digital video cameras are shown as examples of applications of the image processing device, but the present invention is not limited to these. The present invention can be applied to various electronic devices equipped with an imaging element.

[0011] [First Example] 1 is a block diagram showing an example of the basic configuration of an image pickup device 100 according to this embodiment. Although an example of a signal processing device in the image pickup device 100 is shown, signal processing may also be performed by a personal computer, an application server, or the like.

[0012] The control unit 101 includes, for example, a CPU (Central Processing Unit), and controls all the components included in the imaging device 100. The control unit 101 reads a control program from a ROM (Read Only Memory) 102, loads it into a RAM (Random Access Memory) 103, and executes it.

[0013] The ROM 102 is a non-volatile memory that can be electrically erased and recorded. The ROM 102 stores operation programs for each component included in the image capture device 100 and data such as parameters required for operation. The RAM 103 is a rewritable volatile memory. The RAM 103 is used, for example, to expand programs executed by the CPU and temporarily store data generated during operation of each component included in the image capture device 100.

[0014] The optical system 104 is an imaging optical system having lenses, an aperture, etc. The optical system 104 includes a group of lenses including a zoom lens, a focus lens, etc., and forms an image of light from a subject on the imaging surface of an imaging element.

[0015] The imaging unit 105 includes an imaging element. The imaging element is, for example, a CCD image sensor, a CMOS image sensor, or a SPAD image sensor. The imaging element performs photoelectric conversion on the optical image formed on the imaging surface by the optical system 104, and outputs an analog image signal to the A / D conversion unit 106.

[0016] The A / D conversion unit 106 converts the analog image signal input from the imaging unit 105 into digital image data. The digital image data output from the A / D conversion unit 106 is temporarily stored in the RAM 103.

[0017] The image processing unit 107 performs various image processing on the digital image data stored in the RAM 103. Specifically, these processing include pixel correction processing, demosaicing processing, white balance correction processing, gamma processing, etc., and executes development processing, display processing, recording processing, etc. of the digital image data. In pixel correction processing, processing such as interpolation and complementation is performed on a pixel of interest that is the correction target. This includes correction processing of a pixel of interest (hereinafter also referred to as a defective pixel), which is a specific pixel caused by dark current in the image sensor, manufacturing factors, etc.

[0018] A recording unit 108 performs processing for recording image data, etc. on a built-in recording medium. A communication unit 109 wirelessly connects to an external device (not shown) and performs processing for communicating image data, etc.

[0019] A characteristic configuration of this embodiment will be described with reference to FIG. 2. FIG. 2 is a block diagram showing an example configuration of a defective pixel correction processing unit 200, one of the various image processes performed by the image processing unit 107. In the following, the input image signal to the correction processing unit 200 will be referred to as SIG_I, and the output image signal from the correction processing unit 200 will be referred to as SIG_O. For example, SIG_I and SIG_O are image signals with a Bayer array. The correction processing unit 200 also acquires and processes small region coordinate data and defective pixel coordinate data. The small region coordinate data has position information for small regions, which will be described later, and the defective pixel coordinate data has position information for defective pixels. The small region coordinate data and defective pixel coordinate data will be described later with reference to FIG. 3.

[0020] The correction processing unit 200 includes a small region extraction unit 201. A small region is a partial region that constitutes part of an image captured by an image sensor. SIG_I and small region coordinate data are input to the small region extraction unit 201. The output of the small region extraction unit 201 and defective pixel coordinate data are input to the defective pixel interpolation unit 202. A color interpolation unit 203 performs color interpolation processing on the output of the defective pixel interpolation unit 202. A downsampling unit 204 and an upsampling unit 205 are arranged downstream of the color interpolation unit 203. A color synthesis unit 206 performs synthesis processing on image signals of each color component on the output of the upsampling unit 205. A defective pixel synthesis unit 207 performs pixel synthesis processing on the output of the small region extraction unit 201 based on the defective pixel coordinate data and the output of the color synthesis unit 206. A small region synthesis unit 208 performs synthesis processing on SIG_I based on the small region coordinate data and the output of the defective pixel synthesis unit 207, and outputs SIG_O as the processing result. Details of the processing performed by each unit will be described later.

[0021] Next, the small area coordinate data and the defective pixel coordinate data will be described with reference to Fig. 3. Fig. 3 is a schematic diagram for explaining the defective pixel detection method executed by the image processing unit 107 and the method for setting the small area coordinates and the defective pixel coordinates. To detect defective pixels, an image acquired by imaging in a light-blocking state is used.

[0022] Figure 3(A) shows pixel values ​​of a partial region of an image captured under light-blocking conditions. Of the multiple pixels, pixels whose pixel values ​​are equal to or greater than a threshold value are designated as defective pixels. Furthermore, among the multiple defective pixels, a defective pixel whose pixel value is greater than any of its neighboring pixels is further designated as a representative defective pixel.

[0023] Figure 3(B) shows the state in which the threshold value is set to 15 and defective pixels and representative defective pixels are designated, as compared to Figure 3(A). Pixels 301 shown with a shaded area and whose pixel value is equal to or greater than the threshold value are designated as defective pixels. Pixels 302 shown with diagonal lines are defective pixels and have also been designated as representative defective pixels. Figure 3(B) shows an example of a cross-shaped group of defective pixels centered on the representative defective pixel.

[0024] The defective pixel coordinate data is data that sets the coordinates of all defective pixels, including coordinate data of pixels designated as defective pixels and representative defective pixels. The small area coordinate data is data that sets the coordinates of the upper left pixel when the small area extraction unit 201 extracts a small area including a representative defective pixel. Note that the small area is assumed to be of a specific size and is extracted so that the coordinates of the representative defective pixel are included in the specific coordinates.

[0025] Specifically, the area 303 enclosed by the dotted line frame in Figure 3(B) is the small area. Area 303 indicates a small area obtained by extracting seven pixels (49 pixels) in both the horizontal and vertical directions, with representative flaw pixel 302 at the center. Pixel 304 indicates the pixel at the top left corner of small area 303. In this case, the coordinates of pixel 304 are set as the small area coordinates. Small area coordinate data is set for all representative flaw pixels.

[0026] 2 and 3, the components constituting the correction processing unit 200 will be described. The small area extraction unit 201 receives SIG_I as input and performs a process of extracting a small area based on the small area coordinate data. Specifically, a process of extracting an area of ​​a predetermined size is executed, with the small area coordinates being the coordinates of the upper left corner. In this embodiment, a range of seven pixels is extracted in both the horizontal and vertical directions. Since the small area is set so that it is extracted with the representative defective pixel at its center, the small area includes the representative defective pixel. The small area may also include other defective pixels that are peripheral pixels of the representative defective pixel.

[0027] The defective pixel interpolation unit 202 performs a process of interpolating defective pixels on the output image signal of the small region extraction unit 201 based on the position information of the defective pixels. Specifically, the defective pixel interpolation unit 202 first identifies the position of the defective pixel in the output image signal of the small region extraction unit 201 based on the defective pixel coordinate data. Next, the defective pixel interpolation unit 202 performs a process of selecting a pixel to be used in calculating an interpolated value for the pixel of interest (defective pixel). The pixel selection process will be specifically described with reference to FIG. 4.

[0028] FIG. 4A shows pixels used to calculate an interpolated value when the pixel of interest is an R pixel with respect to the constituent colors R (red), G (green), and B (blue) of the Bayer array. The pixel of interest 401a is a defective pixel. The shaded pixel 402a is a pixel used to calculate an interpolated value of the same color as the pixel of interest 401a. If the pixel 402a is a defective pixel, the defective pixel interpolation unit 202 performs processing to exclude the pixel 402a from the pixels used to calculate the interpolated value, thereby achieving pixel interpolation of higher quality. Even when the pixel of interest is a B pixel with respect to the constituent colors R, G, and B of the Bayer array, processing is performed to select a pixel having a positional relationship similar to that of the pixel 402a relative to the pixel of interest 401a.

[0029] FIG. 4B shows pixels used to calculate an interpolated value when the pixel of interest is a G pixel with respect to the constituent colors R, G, and B of the Bayer array. The pixel of interest 401b is a defective pixel. The shaded pixel 402b is a pixel used to calculate an interpolated value of the same color as the pixel of interest 401b. Note that FIG. 4 shows only one example of how to select pixels to use in calculating an interpolated value; pixel selection can be performed using any method from the pixels surrounding the pixel of interest.

[0030] Next, the defective pixel interpolation unit 202 calculates an interpolated value based on the pixel values ​​of the selected surrounding pixels and replaces the pixel value of the target pixel with the interpolated value. Methods for calculating the interpolated value include, for example, calculating the median pixel value of the selected surrounding pixels or calculating the average pixel value. The defective pixel interpolation unit 202 selects pixels to be used for interpolation, calculates the interpolated value, and replaces the pixel values ​​of the defective pixels with the interpolated value for all defective pixels in the output image signal of the small region extraction unit 201.

[0031] The color interpolation unit 203 performs color separation and interpolation processing on the Bayer array image signal output from the defective pixel interpolation unit 202. In the color separation processing, for example, the image signal to be processed is separated into R component, G component, and B component signals. In the interpolation processing, pixel values ​​are interpolated for pixels at coordinates where no R, G, or B color component signals exist.

[0032] Specific processing performed by the color interpolation unit 203 will be described with reference to FIG. 5. FIG. 5(A) is a schematic diagram illustrating a portion of the image signal of the Bayer array input to the color interpolation unit 203. The color interpolation unit 203 separates the image signal of the Bayer array into signals of R, G, and B components. FIG. 5(B) shows an array corresponding to the signals of each color component after separation. The color interpolation unit 203 inserts zero as the pixel value for pixels at coordinates where no signal of each color component exists. Next, the color interpolation unit 203 performs low-pass filter processing on each color signal. FIG. 5(C) shows an array corresponding to the signals of each color component after low-pass filter processing. The color interpolation unit 203 interpolates the pixel value of the pixel into which zero has been inserted based on the pixel values ​​of surrounding pixels.

[0033] 2 downsamples, at a predetermined reduction ratio, each of the R, G, and B component signals output from the color interpolation unit 203. The predetermined reduction ratio is, for example, 1 / 2 or 3 / 4 in the horizontal and vertical directions of the image.

[0034] The upsampling unit 205 upsamples the R, G, and B component signals output from the downsampling unit 204 at a predetermined magnification ratio by inference using a neural network. The predetermined magnification ratio is a magnification ratio at which the image size related to the output image signal of the upsampling unit 205 becomes the same as the image size related to the input image signal of the downsampling unit 204. For example, if the reduction ratio in the downsampling unit 204 is 1 / 2 in the horizontal and vertical directions, the magnification ratio in the upsampling unit 205 is 2 in the horizontal and vertical directions.

[0035] In this embodiment, the neural network (hereinafter also referred to as NN) used for upsampling is a network that has undergone machine learning based on a reference image and training images. Machine learning is performed in advance based on training images generated by downsampling the reference image at the same reduction rate as that used by the downsampling unit 204. Upsampling processing can be performed at a predetermined enlargement rate by calculation using the network model after machine learning. The image size of the reference image may be the same as the image size related to the output image signal from the small region extraction unit 201. Furthermore, the training image is an image in which the influence of degradation due to artificial defective pixels is added to the reference image at the same positions as the pixels in the output image from the small region extraction unit 201. The training image can be generated through the same processing as that used by the defective pixel interpolation unit 202, color interpolation unit 203, and downsampling unit 204. This enables the upsampling unit 205 to perform higher-quality upsampling by suppressing the influence of degradation due to defective pixels.

[0036] One example of an upsampling method is to apply a machine-learned neural network model to R, G, and B component signals. One method performs upsampling through calculations using a machine-learned neural network model with the R, G, and B component image signals as input and output signals. This method uses a network model that has undergone machine learning, with the target image and training images being treated as three images corresponding to the R, G, and B components, respectively. This makes it possible to perform higher-quality upsampling that takes into account the correlation between each color component.

[0037] 2 combines the output image signals of the R, G, and B components from the upsampling unit 205 and outputs an image signal in a Bayer pattern. Specific processing performed by the color combining unit 206 will be described with reference to FIG. 6.

[0038] Fig. 6(A) is a schematic diagram illustrating a portion of the input image signals of the R, G, and B components to the color synthesis unit 206. The shaded pixels in Fig. 6(A) are pixels extracted by the color synthesis unit 206 for each color component from the input image signal. Fig. 6(B) is a schematic diagram illustrating a portion of the image signal in the Bayer array after color synthesis processing. The color synthesis unit 206 generates an image signal in the Bayer array by allocating the pixel values ​​of the pixels of each extracted color component to coordinates corresponding to each color component in a single image signal.

[0039] 2 combines the output image signal of the color composition unit 206 with the output image signal of the small region extraction unit 201. Specifically, the defective pixel composition unit 207 references the defective pixel coordinate data, and if the coordinate values ​​of the pixel to be processed match the coordinate values ​​of the defective pixel, it outputs the output image signal of the color composition unit 206. On the other hand, the defective pixel composition unit 207 references the defective pixel coordinate data, and if the coordinate values ​​of the pixel to be processed do not match the defective pixel coordinates, it outputs the output image signal of the small region extraction unit 201. This makes it possible to replace only the defective pixels in the output image signal of the small region extraction unit 201 with the output image signal of the color composition unit 206.

[0040] The defective pixel synthesis unit 207 may further perform a weighted addition of the output image signal of the small region extraction unit 201 and the output image signal of the color synthesis unit 206 for the coordinates of adjacent pixels adjacent to the defective pixel, and output the result. For example, synthesis processing using weighted addition is performed with pixel values ​​related to the output image signal of the small region extraction unit 201 at a ratio of p% and pixel values ​​related to the output image signal of the color synthesis unit 206 at a ratio of (1-p)%. The condition "0≦p≦100" is satisfied, and the value of p is, for example, 50. Regarding the pixel values ​​of defective pixels, the effect of spatial signal switching that may occur when pixel values ​​related to the output image signal of the small region extraction unit 201 are replaced with pixel values ​​related to the output image signal of the color synthesis unit 206 can be suppressed.

[0041] The small region synthesis unit 208 shown in FIG. 2 synthesizes the input SIG_I with the output image signal of the defective pixel synthesis unit 207 at the corresponding coordinates. Specifically, the small region synthesis unit 208 references the small region coordinate data, and if the coordinate values ​​of the region of interest match the coordinate values ​​of the small region, outputs the output image signal of the defective pixel synthesis unit 207. The small region synthesis unit 208 also references the small region coordinate data, and if the coordinate values ​​of the region of interest do not match the coordinate values ​​of the small region, outputs SIG_I. This allows only the pixel values ​​in the small region of SIG_I to be replaced with pixel values ​​based on the output image signal of the defective pixel synthesis unit 207. A small region is an area that is the same size as the small region extracted by the small region extraction unit 201 and whose small region coordinates are the coordinates of the pixel at the upper left corner. This makes it possible to reflect the processing results in the small region from the defective pixel interpolation unit 202 to the defective pixel synthesis unit 207 on SIG_I before the small region is extracted.

[0042] In this embodiment, downsampling is performed after interpolation of the pixel of interest, and then upsampling is performed by inference using a neural network. According to this embodiment, it is possible to perform higher quality correction on the pixel of interest.

[0043] [Second Example] Next, a second embodiment will be described with reference to FIGS. 7 to 10. In this embodiment, an example is shown in which the downsampling reduction ratio and the upsampling expansion ratio in the correction process for defective pixels are changed according to predetermined conditions. In this embodiment, the same components as those in the first embodiment will be omitted from detailed description by using the same reference numerals as those in the first embodiment, and differences will be mainly described. This method of omitting explanations will also be used in the embodiments described later.

[0044] FIG. 7 is a block diagram showing an example of the configuration of the defective pixel correction processing unit 200 in this embodiment. The difference from the configuration shown in FIG. 2 is that a magnification calculation unit 701 has been added. The magnification calculation unit 701 calculates the reduction ratio of the downsampling unit 204 and the enlargement ratio of the upsampling unit 205 based on the output image signal of the small region extraction unit 201. Specifically, the magnification calculation unit 701 first calculates the average luminance related to the output image signal of the small region extraction unit 201. Next, the magnification calculation unit 701 calculates the degree of degradation in image quality due to the defective pixel based on the average luminance. This will be described in detail with reference to FIG. 8.

[0045] FIG. 8 is a graph showing an example of the relationship between the average luminance (horizontal axis) and the degree of image quality degradation (vertical axis). Image quality degradation due to defective pixels tends to be more noticeable the higher the contrast between the defective pixel and its surrounding pixels, and less noticeable the lower the contrast. FIG. 8 illustrates the relationship between the average luminance and the degree of image quality degradation when a defective pixel is assumed to be caused by dark current. In this case, the darker the surrounding pixels are, the higher the contrast of the defective pixel, making the degradation more noticeable. Conversely, the brighter the surrounding pixels are, the lower the contrast, making the degradation less noticeable. Therefore, the magnification calculation unit 701 calculates the degree of image quality degradation so that the higher the average luminance, the smaller the degree of image quality degradation. The magnification calculation unit 701 determines the reduction ratio of the downsampling unit 204 and the expansion ratio of the upsampling unit 205 based on the calculated degree of image quality degradation. FIG. 9 shows an example of a method for referencing table data.

[0046] Fig. 9 is a table showing an example of table data, showing reduction ratios and enlargement ratios corresponding to the degree of image quality degradation. In the example of Fig. 9, as the degree of image quality degradation increases, the reduction ratio of the downsampling unit 204 tends to increase, and the enlargement ratio of the upsampling unit 205 tends to decrease. The reduction ratio and enlargement ratio are inversely proportional to each other. Note that Fig. 9 shows an example of three levels, but it is possible to expand this to two levels, four levels or more, or to continuously change the reduction ratio and enlargement ratio according to the degree of image quality degradation.

[0047] The effect of correcting defective pixels by the upsampling unit 205 tends to be stronger the larger the upsampling magnification ratio, and weaker the smaller the magnification ratio. On the other hand, adverse effects due to reduced resolution and correction marks tend to be greater the larger the magnification ratio, and tend to be smaller the smaller the magnification ratio. Therefore, in this embodiment, by changing the magnification ratio according to the degree of image quality degradation, it is possible to achieve a correction effect appropriate to the degree to which the degradation of defective pixels is noticeable, and also to suppress adverse effects due to overcorrection.

[0048] The downsampling unit 204 acquires data of the reduction ratio determined by the magnification calculation unit 701 and performs downsampling according to the reduction ratio. The upsampling unit 205 acquires data of the enlargement ratio determined by the magnification calculation unit 701 and performs upsampling according to the enlargement ratio. For example, the upsampling unit 205 includes machine-learned models of multiple neural networks with different enlargement ratios, and switches the machine-learned model to be used depending on the enlargement ratio.

[0049] In this embodiment, the degree of image quality degradation corresponding to the average luminance is calculated, and the reduction ratio and enlargement ratio corresponding to the degree of image quality degradation are calculated. Without being limited to this example, the magnification calculation unit 701 may calculate the reduction ratio and enlargement ratio directly from the average luminance without first calculating the degree of image quality degradation. This process makes it possible to achieve a correction effect appropriate to the degree of visibility of the degradation of defective pixels, and also to suppress the adverse effects of overcorrection.

[0050] In this embodiment, the magnification calculation unit 701 calculates the degree of image quality degradation from the average luminance of the output image signal from the small region extraction unit 201, but any method may be used as long as it calculates the degree of image quality degradation based on the luminance of pixels surrounding a defective pixel. For example, the magnification calculation unit 701 calculates the degree of image quality degradation from the ratio between the pixel value of the defective pixel and the pixel values ​​of pixels surrounding the defective pixel.

[0051] The method of calculating the degree of image quality degradation in the magnification calculation unit 701 is not limited to a calculation method based on the luminance of pixels surrounding the defective pixel. For example, the magnification calculation unit 701 calculates the degree of image quality degradation based on the temperature of the image sensor when acquiring the input image signal SIG_I. In this case, a temperature detection unit for the image sensor is provided, and the temperature detection unit measures the temperature using an image sensor thermometer (not shown). The control unit 101 in FIG. 1 acquires temperature information of the image sensor from the temperature detection unit and outputs the temperature information of the image sensor to the magnification calculation unit 701. The magnification calculation unit 701 calculates the degree of image quality degradation based on the temperature information of the image sensor. Referring to FIG. 10, an example of calculating the degree of image quality degradation from the temperature information of the image sensor is shown.

[0052] FIG. 10 is a graph showing an example of the relationship between the temperature of the image sensor (shown on the horizontal axis) and the degree of image quality degradation (shown on the vertical axis). In FIG. 10, the degree of degradation due to scratches caused by dark current is assumed. The magnification calculation unit 701 performs calculation processing so that the higher the temperature of the image sensor, the greater the degree of image quality degradation. Regarding the influence of scratches caused by dark current, the higher the temperature of the image sensor, the greater the signal amplitude, making the degradation more noticeable. Furthermore, if the image sensor is a SPAD image sensor that utilizes the avalanche emission phenomenon, the signal amplitude related to the defective pixel may be larger. If the influence of the defective pixel spreads to surrounding pixels due to crosstalk, the degradation becomes even more noticeable. Therefore, as shown in FIG. 10, the magnification calculation unit 701 calculates a greater degree of image quality degradation the higher the temperature of the image sensor when acquiring SIG_I. Another method for calculating the degree of image quality degradation that assumes scratches caused by dark current is to calculate the degree of image quality degradation according to the exposure time when acquiring SIG_I by the image sensor. In this case, the exposure time is used instead of the temperature of the image sensor in FIG. 10. Although FIG. 10 shows a graph of linear characteristics as an example, the degree of image quality degradation may be calculated according to a graph of non-linear characteristics or characteristics expressed by a step function.

[0053] In this embodiment, the downsampling reduction ratio and the upsampling enlargement ratio in the correction process for the pixel of interest are changed according to the degree of image quality degradation, thereby making it possible to perform higher quality correction on the pixel of interest.

[0054] [Third Example] A third embodiment will be described with reference to Fig. 11 and Fig. 12. This embodiment shows an example in which the reliability of the correction result of a defective pixel is determined and the defective pixel is corrected. Fig. 11 is a block diagram showing an example of the configuration of a defective pixel correction processing unit 200 in this embodiment. The difference from the configuration shown in Fig. 2 is that a correction value generation unit 1101, a reliability determination unit 1102, and a low-reliability correction unit 1103 are provided.

[0055] The correction value generation unit 1101 generates a correction value for the output image signal of the small region extraction unit 201 based on the defective pixel coordinate data. If the reliability of the correction value for a defective pixel generated by a series of processes from the defective pixel interpolation unit 202 to the color synthesis unit 206 is low (low reliability), the correction value generation unit 1101 generates a correction value for the defective pixel to be used as a substitute. Specifically, the correction value generation unit 1101 identifies the defective pixel using the defective pixel coordinate data for the output image signal of the small region extraction unit 201. Next, the correction value generation unit 1101 selects surrounding pixels to be used to calculate an interpolated value for the defective pixel. The method of selecting pixels to be used to calculate the interpolated value is the same as the method described for the defective pixel interpolation unit 202. Next, the correction value generation unit 1101 calculates an interpolated value based on the pixel values ​​of the selected surrounding pixels, and replaces the pixel value of the pixel of interest with the interpolated value. Methods for calculating the interpolated value include, for example, calculating the median pixel value of the selected surrounding pixels or calculating the average pixel value. The correction value generation unit 1101 performs the process of selecting pixels to be used for interpolation, the process of calculating interpolated values, and the process of replacing the pixel values ​​of defective pixels with interpolated values ​​for all defective pixels related to the output image signal of the small region extraction unit 201.

[0056] The reliability determination unit 1102 acquires the output image signal of the color synthesis unit 206 and the output image signal of the small region extraction unit 201 and performs reliability determination based on the defective pixel coordinate data. The output image signal of the color synthesis unit 206 is compared with the output image signal of the small region extraction unit 201 to determine the reliability of the correction values ​​of the defective pixels generated by a series of processes from the defective pixel interpolation unit 202 to the color synthesis unit 206. Specifically, the reliability determination unit 1102 identifies pixels in the output image signal of the color synthesis unit 206 that are not defective pixels (hereinafter referred to as non-defective pixels) based on the defective pixel coordinate data. Next, the reliability determination unit 1102 calculates the differences between corresponding pixels between the output image signal of the color synthesis unit 206 and the output image signal of the small region extraction unit 201 for all non-defective pixels, and further calculates the average value of the differences for the non-defective pixels. Next, the reliability determination unit 1102 calculates the reliability of the correction of the defective pixels from the average value of the differences for the non-defective pixels.

[0057] The reliability calculation process for defective pixel correction will be described with reference to Fig. 12. Fig. 12 is a graph showing an example of the relationship between the average value of the differences of non-defective pixels shown on the horizontal axis and the reliability of defective pixel correction shown on the vertical axis.

[0058] The reliability of defective pixel correction will be described by focusing on the processing performed by the upsampling unit 205, which is part of the series of processing from the defective pixel interpolation unit 202 to the color synthesis unit 206. Regarding the inference results of the upsampling using a neural network in the upsampling unit 205, the most likely result is a state in which the pixel values ​​of defective and non-defective pixels are inferred to be close to those of the original object image. Here, the pixel values ​​of the original object image for non-defective pixels are those that match the pixel values ​​of the image corresponding to the output image signal from the small region extraction unit 201. However, in special situations, such as when the object has a specific pattern, the upsampling unit 205 may generate a pattern that does not exist in the original object image through inference. In this case, both the defective pixels corrected in the output image signal from the upsampling unit 205 and the non-defective pixels are likely to differ from the original object image. In this embodiment, the reliability of the inference for defective pixels can be indirectly estimated by referring to the difference between the output image signal from the color synthesis unit 206 and the output image signal from the small region extraction unit 201 for non-defective pixels.

[0059] For the above reasons, the reliability determination unit 1102 calculates the reliability of the defective pixel correction so that the greater the average value of the differences of the non-defective pixels, the lower the reliability of the defective pixel correction. In the example of FIG. 12, when the average value of the differences of the non-defective pixels is equal to or less than the first threshold, the reliability of the defective pixel correction is constant. When the average value of the differences of the non-defective pixels is greater than the first threshold and equal to or less than the second threshold, the reliability of the defective pixel correction decreases as the average value of the differences of the non-defective pixels increases. When the average value of the differences of the non-defective pixels is greater than the second threshold, the reliability value of the defective pixel correction is zero. The reliability determination unit 1102 outputs the determination result as reliability information to the low-reliability correction unit 1103.

[0060] The low-reliability correction unit 1103 shown in FIG. 11 acquires the output image signal of the color synthesis unit 206 and the output image signal of the correction value generation unit 1101, and performs correction processing based on reliability information of the defective pixel correction output from the reliability determination unit 1102. Using the reliability information of the defective pixel correction, a synthesis process is performed on the output image signal of the color synthesis unit 206 and the output image signal of the correction value generation unit 1101. Specifically, the lower the reliability of the defective pixel correction, the smaller the value of the synthesis ratio set by the low-reliability correction unit 1103 for the output image signal of the color synthesis unit 206, and the larger the value of the synthesis ratio set by the correction value generation unit 1101 for the output image signal. The synthesis ratio corresponds to a weighting coefficient and represents the proportion of synthesis for the output image signal. The low-reliability correction unit 1103 performs weighted addition on multiple output image signals according to the set synthesis ratio.

[0061] The correction method used by the correction value generation unit 1101 is a defective pixel correction method that does not use a neural network. This method may result in lower quality defective pixel correction than the series of processes from the defective pixel interpolation unit 202 to the color synthesis unit 206. However, this method has the advantage that the defective pixel correction result is less likely to deviate from the likely correction result. Therefore, when the reliability of the defective pixel correction performed by the series of processes from the defective pixel interpolation unit 202 to the color synthesis unit 206 is low, it is desirable to use the image signal output by the correction value generation unit 1101. This makes it possible to modify the defective pixel correction result when there is a possibility that the defective pixel correction result will deviate from the likely correction result.

[0062] The defective pixel synthesis unit 207 acquires the output image signal of the small region extraction unit 201 and the output image signal of the low-reliability correction unit 1103, and executes synthesis processing based on the defective pixel coordinate data. In this embodiment, processing is performed to synthesize the output image signal of the small region extraction unit 201 and the output image signal of the low-reliability correction unit 1103.

[0063] The reliability determination unit 1102 of this embodiment calculates the reliability based on the average value of the differences between non-defective pixels in the image signal output by the color synthesis unit 206 and the image signal output by the small region extraction unit 201. This is not a limitation, and any reliability calculation method may be used as long as it involves comparing non-defective pixels. For example, the reliability determination unit 1102 may calculate the reliability based on the maximum value of the differences between non-defective pixels in the image signal output by the color synthesis unit 206 and the image signal output by the small region extraction unit 201. This also makes it possible to estimate the likelihood of defective pixel correction.

[0064] Furthermore, the low-reliability correction unit 1103 may combine the output image signal of the defective pixel interpolation unit 202 and the output image signal of the color synthesis unit 206, instead of the output image signal of the correction value generation unit 1101. This also makes it possible to modify the correction results of the defective pixels using a signal that has been subjected to correction of the defective pixels without using the inference results of the NN.

[0065] According to this embodiment, the reliability of the correction result of the defective pixel is determined, and the defective pixel can be corrected with higher quality based on the result of the reliability determination.

[0066] Although the preferred embodiments of the present invention have been described, the present invention is not limited to these, and various modifications and variations are possible within the scope of the present invention. Parts of the configurations of the above-described embodiments may be combined as appropriate.

[0067] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0068] 100 Imaging device 101 Control section 105 Imaging unit 107 Image processing section 200 Correction processing section

Claims

1. an acquisition means for acquiring position information of a pixel of interest in an image; extraction means for extracting an area including the pixel of interest from the image; an interpolation unit that performs interpolation for the pixel of interest in the region extracted by the extraction unit; downsampling the image having the pixels interpolated by the interpolation means; a first processing means for outputting data of a first image; a second processing means for performing upsampling on the data of the first image by a calculation based on machine learning using a correct image and training images to generate data of a second image; a synthesis means for synthesizing a pixel of interest in the region with a corresponding pixel in the second image to generate data of a third image.

1. An image processing device comprising:

2. the pixel of interest is a pixel whose pixel value is equal to or greater than a threshold value in an image captured by an imaging element in a light-shielded state, The synthesizing means synthesizes pixels in the second image that are peripheral pixels of the pixel of interest in the region and have pixel values ​​equal to or greater than a threshold value with corresponding pixels in the second image.

2. The image processing device according to claim 1, wherein:

3. The training image is an image obtained by adding the influence of degradation due to pseudo defective pixels at a position corresponding to the pixel of interest in the region to the correct image.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. the region has predetermined coordinates for the position of the pixel of interest; The size of the correct image is the same as the size of the region, The training image is an image to which the influence of the degradation is added at a position corresponding to the coordinate in the correct image.

4. The image processing device according to claim 3.

5. a calculation unit that calculates the reduction ratio of the first processing unit and the enlargement ratio of the second processing unit based on the image of the region extracted by the extraction unit; 5. The image processing device according to claim 1, wherein the image processing device is a computer.

6. the calculation means calculates the reduction ratio and the enlargement ratio based on a degree of image quality degradation caused by the pixel of interest; The first processing means performs downsampling at a first reduction rate when the degradation level is a first degradation level, and performs downsampling at a second reduction rate when the degradation level is a second degradation level that is greater than the first degradation level.

6. The image processing device according to claim 5,

7. The calculation means calculates the degree of deterioration from an average luminance of an image related to the region, or a ratio between the pixel value of the pixel of interest and the pixel values ​​of pixels surrounding the pixel of interest.

7. The image processing device according to claim 6,

8. The calculation means calculates the deterioration degree from temperature information acquired by a temperature detection means of the image pickup element.

7. The image processing device according to claim 6,

9. The calculation means calculates the degree of deterioration from the exposure time of the image sensor.

7. The image processing device according to claim 6,

10. an acquisition means for acquiring position information of a pixel of interest in an image; extraction means for extracting an area including the pixel of interest from the image; an interpolation unit that performs interpolation for the pixel of interest in the region extracted by the extraction unit; downsampling the image having the pixels interpolated by the interpolation means; a first processing means for outputting data of a first image; a second processing means for performing upsampling on the data of the first image by a calculation based on machine learning using a correct image and training images to generate data of a second image; a correction means for correcting the pixel of the second image in accordance with the reliability information of the correction of the pixel of interest, and generating data of a third image; a synthesis means for synthesizing a pixel of interest in the region with a corresponding pixel in the third image to generate data of a fourth image.

1. An image processing device comprising:

11. a determination means for determining the reliability of the correction of the pixel of interest; a correction value generating means for generating a correction value for the pixel of interest using the position information from the output of the extracting means, the determining means calculates a difference between a pixel in the region that is not the pixel of interest and a pixel in the second image that corresponds to the pixel of interest, and determines reliability; The correction means generates the third image data by combining the second image data and the image data output by the correction value generation means in accordance with the reliability determination result by the determination means.

11. The image processing device according to claim 10.

12. the determination means calculates a reliability based on an average value or a maximum value of the differences; The correction means when the determination means calculates a first reliability, a value of a combination ratio for the second image is set to a first value, and a value of a combination ratio for the image output by the correction value generation means is set to a second value; When the determination means calculates a second reliability lower than the first reliability, the value of the combination ratio for the second image is set to a third value smaller than the first value, and the value of the combination ratio for the image output by the correction value generation means is set to a fourth value larger than the second value.

12. The image processing device according to claim 11.

13. a determination means for determining the reliability of the correction of the pixel of interest, the determining means calculates a difference between a pixel in the region that is not the pixel of interest and a pixel in the second image that corresponds to the pixel of interest, and determines reliability; The correction means generates the third image data by combining the second image data and the image data output by the interpolation means in accordance with the reliability determination result by the determination means.

11. The image processing device according to claim 10.

14. The correction value generating means generates the correction value from pixel values ​​of a plurality of pixels selected from pixels surrounding the pixel of interest.

12. The image processing device according to claim 11.

15. The interpolation means calculates an interpolated value based on pixel values ​​of a plurality of pixels selected from pixels surrounding the pixel of interest.

14. The image processing device according to claim 13.

16. An image processing device according to any one of claims 1 to 15 is provided. An imaging device characterized by:

17. Equipped with an imaging element that utilizes the avalanche light emission phenomenon 17. The imaging device according to claim 16.

18. An image processing method executed by an image processing device that performs pixel correction, an acquisition step of acquiring position information of a pixel of interest in an image; an extraction step of extracting a region including the pixel of interest from the image; an interpolation step of performing interpolation for the pixel of interest in the extracted region; a first processing step of downsampling the image having interpolated pixels to output data of a first image; a second processing step of upsampling the first image data by a machine learning-based operation using a correct answer image and training images to generate second image data; a combining step of combining a pixel of interest in the region with a corresponding pixel in the second image to generate data of a third image. An image processing method comprising:

19. An image processing method executed by an image processing device that performs pixel correction, an acquisition step of acquiring position information of a pixel of interest in an image; an extraction step of extracting a region including the pixel of interest from the image; an interpolation step of performing interpolation for the pixel of interest in the extracted region; a first processing step of downsampling the image having interpolated pixels to output data of a first image; a second processing step of upsampling the first image data by a machine learning-based operation using a correct answer image and training images to generate second image data; a correction step of correcting pixels of the second image in accordance with the reliability information of the correction of the pixel of interest, and generating data of a third image; a combining step of combining a pixel of interest in the region with a corresponding pixel in the third image to generate data of a fourth image. An image processing method comprising:

20. 20. A program for causing a computer of an image processing apparatus to execute each step according to claim 18 or 19.

Images