Image processing device, imaging device, image processing method, program and recording medium
Patent Information
- Application Number
- JP2022036302
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2022-03-09
- Publication Date
- 2025-03-13
- Estimated Expiration
- 2042-03-09
AI Technical Summary
Existing image processing techniques that add uniform noise to the entire image area result in noise unevenness and resolution unevenness, which degrade image quality.
A method of generating composite images using a plurality of reference images with varying composition ratios, where the average composition ratio for each image block is adjusted to minimize noise and resolution unevenness by averaging the composition ratios across multiple images.
This approach produces high-quality images with reduced noise unevenness and resolution unevenness by optimizing the composition ratios across image blocks, resulting in improved image quality.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to an image processing device, and more particularly to an image processing device that combines a plurality of images. [Background technology]
[0002] It is known that there are conventional image processing methods that reduce noise by combining multiple images. In such processing, taking into consideration situations such as when the subject is moving, there is a technology that divides multiple images into a single standard image that serves as a basis for combining and reference images, and changes the combining ratio between the standard image and each reference image to reduce noise while preventing image multiplexing (Patent Document 1).
[0003] Furthermore, Patent Document 2 discloses a technique for obtaining an image in which the texture of the subject is enhanced by adding a predetermined noise component to an image that has been processed to reduce noise. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-164857 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-28348 Summary of the Invention [Problem to be solved by the invention]
[0005] However, the technology described in Patent Document 2 has a problem in that it adds a uniform amount of noise to the entire image area, which can cause uneven noise and uneven resolution.
[0006] Therefore, an object of the present invention is to provide a technology that reduces noise unevenness and resolution unevenness in an image after noise reduction processing by combining multiple images, while enhancing the texture of the subject. [Means for solving the problem]
[0007] In order to solve the above problem, the present invention provides an image processing device having a synthesis means for synthesizing a second composite image generated using a plurality of first composite images generated based on a reference image and a plurality of reference images different from the reference image, and the reference image to generate a third composite image, wherein when an arithmetic average of synthesis ratios for generating the first composite image is a first value, a synthesis ratio for the reference image when generating the third composite image is a third value, when an arithmetic average of synthesis ratios for generating the first composite image is a second value, a synthesis ratio for the reference image when generating the third composite image is a fourth value, and when the first value is higher than the second value, the third value is higher than the fourth value. [Effects of the Invention]
[0008] According to the present invention, it is possible to obtain a high-quality image while reducing noise variations and resolution variations by combining a plurality of images. [Brief explanation of the drawings]
[0009] [Figure 1] 1 is a block diagram illustrating an image processing apparatus according to an embodiment of the present invention. [Figure 2] 4 is a flowchart illustrating the operation of the image processing device according to the first embodiment of the present invention. [Figure 3] FIG. 2 is a diagram showing a plurality of images captured successively in the time direction in the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing an image divided into small rectangles for multiple-image compositing processing according to the first embodiment of the present invention. [Figure 5] FIG. 3 is a diagram illustrating generation of a first composite image in the embodiment of the present invention. [Figure 6] 5A to 5C are diagrams illustrating generation of a second composite image in the embodiment of the present invention. [Figure 7]FIG. 10 is a block diagram of an image processing device according to a second embodiment of the present invention. [Figure 8] 10 is a flowchart illustrating the operation of an image processing device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION
[0010] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.
[0011] (First embodiment) An image processing device according to a first embodiment of the present invention will be described. The image processing device in this embodiment is, for example, a digital camera for personal use. Fig. 1 is a block diagram for explaining the image processing device in the first embodiment of the present invention.
[0012] The image processing device 1 according to this embodiment comprises an image sensor 101, an image synthesis unit 102, a development processing unit 103, a DMAC (DMA controller) 104, a CPU 105, a bus 106, a DRAM 107, and a medium 108 such as an SD card.
[0013] The image sensor 101 receives light via an optical system (not shown) and converts the optical signal into an electrical signal.
[0014] The image synthesis unit 102 performs synthesis processing on the received images. Specific details will be described later, but in this embodiment, the image synthesis unit 102 receives images from the DMAC 104 and inputs a first synthesis ratio and the first synthesized image to the DMAC 104. The image synthesis unit 102 also receives a plurality of first synthesis ratios and a plurality of first synthesized images from the DMAC 104 and inputs a second synthesis ratio and the second synthesized image to the DMAC 104. The image synthesis unit 102 also receives a second synthesis ratio, the second synthesized image, and a reference image that is one of the plurality of images from the DMAC 104 and inputs an output synthesized image to the DMAC 104.
[0015] The development processing unit 103 performs development processing on the image captured by the image sensor 101 and the output composite image output by the image composition unit 102. Development processing refers to the process of generating data for three colors (red, green, and blue) from each pixel of the image and converting it into a viewable format such as JPG format.
[0016] The DMAC 104 records the image captured by the image sensor 101, the first composition ratio, the first composite image, the second composition ratio, and the second composite image in the DRAM 107 via the bus 106, and reads them from the DRAM 107 and performs processing such as inputting them to the image composition unit 102.
[0017] The CPU 105 controls the image synthesis unit 102, the development processing unit 103, and the DMAC 104 via a bus 106.
[0018] The CPU 105 records the composite image output by the DMAC 104 onto the medium 108 via the bus 106 .
[0019] Next, the operation of the image processing device configured as above will be described with reference to Fig. 2, which is a flowchart illustrating the operation of the image processing device according to this embodiment.
[0020] First, a plurality of images captured by the image sensor 101 and used for image synthesis processing are input to the DMAC 104 , which then records them in the DRAM 107 .
[0021] FIG. 3 is a diagram showing a plurality of images captured successively in the time direction in this embodiment. The plurality of images used in the image synthesis process are a plurality of images 300 captured successively in the time direction as shown in FIG. 3. The subjects in the plurality of images 300 include a subject 301 that remains stationary during image capture and a subject 302 that moves during image capture. The position of the subject 301 that remains stationary during image capture does not change in each of the plurality of images 300, but the position of the subject 302 that moves during image capture differs in each of the plurality of images 300. Note that although the number of images shown in FIG. 3 is three, in the following description of this embodiment, the number of images recorded in DRAM 107 that are used in the image synthesis process will be described as N.
[0022] In step S201, the CPU 105 inputs one of the multiple images recorded in the DRAM 107 as a reference image to the image synthesis unit 102. The reference image is an image that serves as a reference for synthesis among the multiple images used in the image synthesis process. For example, the image captured earliest is set as the reference image. The DMAC 104 reads the reference image from the DRAM 107 and inputs it to the image synthesis unit 102.
[0023] In step S202, CPU 105 inputs the kth reference image, other than the reference image, from among the multiple images recorded in DRAM 107 to image synthesis unit 102. k is a natural number ranging from 1 to N-1, with its initial value being 1. Here, CPU 105 assigns numbers to the reference images other than the reference image in order. The order here may be, for example, the order of the time of capture. DMAC 104 reads out the kth reference image from DRAM 107 and inputs it to image synthesis unit 102.
[0024] In step S203, the image synthesis unit 102 generates a first synthesis ratio using the standard image and one reference image input in steps S201 and S202. The DMAC 104 records the generated first synthesis ratio in the DRAM 107.
[0025] The synthesis process in the image synthesis unit 102 in step S203 is performed by dividing the base image and each reference image into blocks like image 400 shown in Fig. 4, and performing the synthesis process on a block-by-block basis. A block is a small rectangle like the one shown in Fig. 4. It is preferable to divide the blocks into similar sizes.
[0026] In step S204, the image synthesis unit 102 synthesizes the standard image and one reference image input in steps S201 and S202 based on the first synthesis ratio generated in step S203 to generate a first synthesized image. The DMAC 104 records the generated first synthesized image in the DRAM 107.
[0027] The method of generating the first synthesis ratio and the first synthesis image in the image synthesis unit 102 in steps S203 and S204 will be described.
[0028] FIG. 5 is a diagram for explaining generation of a first composite image in this embodiment. For the purpose of explanation, numbers are assigned to blocks of the base image, blocks of the kth reference image, and blocks of a composite image (composite block) of the base image and the kth reference image, which is one of the first composite images. Specifically, as shown in FIG. 5, m is the block number, and names are assigned such as Bm, Rk_m, and Ck_m. m is an integer equal to or greater than 0. Blocks Bm, Rk_m, and Ck_m assigned similar numbers are located at corresponding positions in the image area. Here, corresponding positions refer to positions located at the same coordinates, and if there is a positional deviation between the images, they refer to positions located at the same coordinates after alignment.
[0029] Furthermore, a name such as Ak_m is given to the blending ratio of the kth reference image, which is one of the first blending ratios corresponding to the blocks Bm, Rk_m, and Ck_m.
[0030] The pixel values of the blocks Bm, Rk_m, and Ck_m, and the blending ratio Ak_m, satisfy the following (Equation 1).
[0031]
number
[0032] As the value of the first blending ratio Ak_m increases, the ratio of the pixel values of the block Rk_m to the pixel values of the block Ck_m of the first blended image increases. The blending ratio Ak_m is calculated by (Equation 2).
[0033]
number
[0034] In the above (Equation 2), α is a predetermined adjustment parameter, and α>0. Dk_m is the average value of the differences in pixel values between block Bm and block Rk_m, and is expressed by (Equation 3).
[0035]
number
[0036] In (Equation 3), Bml and Rk_ml are pixel values at corresponding positions in the blocks Bm and Rk_m, respectively, and n is the number of pixels in the blocks Bm and Rk_m.
[0037] 4, for block 401 on a stationary subject, there is no difference between the base image and each reference image, and the value of the first combination ratio Ak_m corresponding to block 401 is maximum. On the other hand, for block 402 on a moving subject, there is a difference between the base image and each reference image, and as this difference increases, the value of the first combination ratio Ak_m corresponding to block 402 decreases.
[0038] As described above, in steps S203 and S204, the image synthesis unit 102 performs synthesis processing on the base image and each reference image. Using the base image and the kth reference image, the image synthesis unit 102 outputs a synthesis ratio for the kth reference image, which is one of the first synthesis ratios, and a synthesized image of the base image and the kth reference image, which is one of the first synthesis images.
[0039] In step S205, if k=N-1 is true, the flow proceeds to step S207, otherwise, in step S206, k is incremented and then the flow returns to step S202.
[0040] In step S207, the DMAC 104 reads out from the DRAM 107 all of the first blending ratios and all of the first blended images generated in steps S203 and S204, and inputs them to the image blending unit .
[0041] In step S208, the image synthesis unit 102 generates a second synthesis ratio using the first synthesis ratio input in step S207. The second synthesis ratio generated by the DMAC 104 is recorded in the DRAM 107.
[0042] In step S208, the image synthesis unit 102 generates a second synthesis ratio by averaging the N-1 first synthesis ratios.
[0043] In step S209, the image synthesis unit 102 generates a second synthesized image using the first synthesized image input in step S207. Specifically, the image synthesis unit 102 generates the second synthesized image by averaging N-1 first synthesized images. The DMAC 104 records the generated second synthesized image in the DRAM 107.
[0044] In step S210, the DMAC 104 reads the reference image, the second blending ratio generated in step S208, and the second blended image generated in step S209 from the DRAM 107, and inputs them to the image blending unit .
[0045] In step S211, the image synthesis unit 102 generates one output synthesis image using the reference image, the second synthesis ratio, and the second synthesis image input in step S210. The DMAC 104 records the generated output synthesis image on the medium 108.
[0046] In step S211, the image synthesis unit 102 synthesizes the second synthesized image and the reference image according to the second synthesis ratio.
[0047] FIG. 6 is a diagram for explaining generation of the second composite image in this embodiment.
[0048] As shown in FIG. 6, the blending amount γm of the block Bm of the reference image relative to the block Cm of the second blended image can be calculated by the following (Equation 4) using the second blending ratio Am_ave.
[0049]
number
[0050] In (Equation 4), β is a predetermined parameter for adjustment, and 0<β<=1.
[0051] A block Cm_out of the output composite image at a position corresponding to the block Cm of the second composite image is determined by (Equation 5).
[0052]
number
[0053] The second combination ratio is the arithmetic average of N-1 first combination ratios. The second combined image is also the arithmetic average of N-1 first combined images. Therefore, the larger the value of the second combination ratio, the closer to uniform the components of the base image and the reference image included in the second combined image. The more uniform the components of the images included in the second combined image, the higher the degree of noise reduction by combination, so the second combination ratio represents the degree of noise reduction in the second combined image. In other words, noise unevenness can be reduced by adjusting the amount of combination of the base image according to the second combination ratio.
[0054] When the output composite image generated by the image composition unit 102 is recorded on the medium 108, the flow ends.
[0055] According to the first embodiment, when combining a plurality of images, it is possible to reduce noise as well as noise unevenness.
[0056] (Second embodiment) An image processing device according to a second embodiment of the present invention will be described. Unlike the first embodiment, the second embodiment performs noise reduction processing again on the second composite image. Fig. 7 is a block diagram for explaining the image processing device according to the second embodiment of the present invention.
[0057] The configuration of the image processing device according to this embodiment shown in Fig. 7 is a configuration in which a noise reduction processing unit 709 is added to the configuration of the image processing device according to the first embodiment. Components having the same last two digits of the reference numerals in Fig. 7 correspond to the last two digits of the reference numerals in Fig. 1 of the first embodiment.
[0058] The noise reduction processing unit 709 receives the second blending ratio and the second blended image from the DMAC 704, and inputs the second blended image that has undergone noise reduction processing and the threshold value for noise reduction processing to the DMAC 704.
[0059] The image synthesis unit 702 receives the second synthesis ratio, the second synthesized image that has been subjected to noise reduction processing, the threshold value for the noise reduction processing, and a reference image that is one of the multiple images from the DMAC 704, and inputs the output synthesized image to the DMAC 704.
[0060] The description of other components in FIG. 7 will be omitted.
[0061] The operation of the image processing device configured as above will now be described with reference to the flowchart of FIG.
[0062] Steps S801 to S809 are the same as steps S201 to S209 in FIG.
[0063] In step S810, the DMAC 704 reads the second blending ratio and the second blended image generated in steps S808 and S809 from the DRAM 707 and inputs them to the noise reduction processing unit 709.
[0064] In step S811, the noise reduction processing unit 709 generates a noise-reduced second composite image and a threshold value for the noise reduction processing using the second blending ratio and the second composite image input in step S810. The generated noise-reduced second composite image and the threshold value for the noise reduction processing are recorded in the DRAM 707 by the DMAC 704.
[0065] In step S811, the noise reduction processing unit 709 performs noise reduction processing on the second composite image in accordance with the second composite ratio. With reference to the blocks of the second composite image and the second composite ratio shown in Fig. 6, the threshold value tm of the noise reduction processing to be performed on the block Cm of the second composite image is calculated using the second composite ratio Am_ave according to (Equation 6).
[0066]
number
[0067] In equation 6, γ is an adjustment parameter, and γ>0. The noise reduction process is a known filter process such as a smoothing filter that uses a pixel to be subjected to noise reduction and pixels surrounding the pixel to be subjected to noise reduction, and a threshold value tm is used to select the surrounding pixels to be used. When the difference between the pixel value of the pixel to be subjected to noise reduction and the pixel values of the pixels surrounding the pixel to be subjected to noise reduction is equal to or less than the threshold value tm, the pixels surrounding the pixel to be subjected to noise reduction are used in the filtering process.
[0068] In step S812, the DMAC 704 reads from the DRAM 707 the reference image, the second synthesis ratio generated in steps S808 and S811, the threshold value for the noise reduction processing, and the second synthesis image after noise reduction processing, and inputs them to the image synthesis unit 702.
[0069] In step S813, the image composition unit 702 generates one output composite image. The DMAC 704 records the generated output composite image on the medium 708.
[0070] In step S813, the image synthesis unit 702 synthesizes the reference image with the second synthesized image that has undergone noise reduction processing.
[0071] A block of the second composite image that has undergone noise reduction processing and that is located at a position corresponding to block Cm of the second composite image is designated as Cm_NR. The blending amount rm of block Bm of the reference image relative to block Cm_NR of the second composite image that has undergone noise reduction processing may be set so as to be positively correlated with the second blending ratio Am_ave.
[0072] A block Cm_out of the output composite image that is located at a position corresponding to a block Cm_NR of the second composite image that has undergone noise reduction processing and a block Bm of the reference image can be calculated using (Equation 7).
[0073]
number
[0074] The flow ends when the output composite image generated by the image composition unit 702 is recorded on the medium 708.
[0075] The above is the description of the second embodiment.
[0076] According to the second embodiment, it is possible to achieve a reduction in noise unevenness compared to the first embodiment, and it is also expected that a further noise reduction effect will be achieved.
[0077] (Other embodiments) In the above embodiment, the explanation was based on a personal digital camera, but it can also be applied to mobile devices, smartphones, or network cameras connected to a server, as long as they are equipped with a synthesis function.
[0078] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and run the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]
[0079] 1. Image processing device 101 Image sensor 102 Image synthesis unit 103 Development processing section 104 DMAC (DMA Controller) 105 CPU Bus 106 107 DRAM 108 Media 7 Image Processing Device 701 Image Sensor 702 Image synthesis unit 703 Development Processing Unit 704 DMAC (DMA Controller) 705 CPU 706 Bus 707 DRAM 708 Media 709 Noise Reduction Processing Unit
Claims
1. a synthesis means for synthesizing a second synthetic image generated using a plurality of first synthetic images generated based on a reference image and a plurality of reference images different from the reference image, and the reference image to generate a third synthetic image, when an arithmetic average of the synthesis ratios for generating the first synthesis image is a first value, a synthesis ratio for generating the third synthesis image with respect to the reference image is a third value; when an arithmetic average of the synthesis ratios for generating the first synthesis image is a second value, a synthesis ratio for generating the third synthesis image with respect to the reference image is a fourth value; 2. An image processing apparatus comprising: when the first value is higher than the second value, the third value is higher than the fourth value.
2. The image processing device according to claim 1, characterized in that the synthesis means performs the synthesis on the reference image and one of the second synthetic images generated using the multiple first synthetic images generated based on one of the reference images and each of the multiple reference images, thereby generating one of the third synthetic images.
3. 3. The image processing apparatus according to claim 1, further comprising a processing unit for performing a process for reducing noise in the second composite image.
4. the processing performed by the processing means is a filter processing, 4. The image processing device according to claim 3, wherein, in the filter process, when a difference between a pixel to be processed and a surrounding pixel of the pixel to be processed is equal to or smaller than a predetermined threshold, the processing means uses the surrounding pixel in processing for noise reduction of the pixel to be processed.
5. 5. The image processing apparatus according to claim 4, wherein the processing means determines the threshold value based on an arithmetic average of synthesis ratios for generating the first synthesis image.
6. 6. The image processing device according to claim 1, wherein a synthesis ratio for generating the first synthesized image is determined based on an average value of differences in pixel values between the standard image and each of the plurality of reference images.
7. 7. The image processing apparatus according to claim 1, wherein the second composite image is generated by averaging the plurality of first composite images.
8. a dividing means for dividing the standard image and the plurality of reference images into blocks, 8. The image processing apparatus according to claim 1, wherein the synthesis means performs the synthesis for the corresponding blocks in the standard image and each of the plurality of reference images.
9. The image processing apparatus according to claim 8 , wherein the corresponding blocks are blocks located at the same coordinates.
10. A first synthesis means for synthesizing a standard image and a reference image different from the standard image at a first synthesis ratio corresponding to each of a plurality of regions of the standard image to generate a first synthetic image; a second synthesis means for synthesizing the reference image with the first synthetic image to generate a second synthetic image, The image processing device is characterized in that the second synthesis means sets a second synthesis ratio, which is a synthesis ratio between the first synthesis image and the reference image in each of a plurality of regions of the reference image, based on the first synthesis ratio.
11. a synthesis step of synthesizing a second synthesized image generated using a plurality of first synthesized images generated based on a reference image and a plurality of reference images different from the reference image, and the reference image to generate a third synthesized image, when an arithmetic average of the synthesis ratios for generating the first synthesis image is a first value, a synthesis ratio for generating the third synthesis image with respect to the reference image is a third value; when an arithmetic average of the synthesis ratios for generating the first synthesis image is a second value, a synthesis ratio for generating the third synthesis image with respect to the reference image is a fourth value; 13. An image processing method, comprising: when the first value is higher than the second value, the third value is higher than the fourth value.
12. A first synthesis step of synthesizing a standard image and a reference image different from the standard image at a first synthesis ratio corresponding to each of a plurality of regions of the standard image to generate a first synthetic image; a second synthesis step of synthesizing the reference image with the first synthetic image to generate a second synthetic image, The image processing method is characterized in that the second synthesis step sets a second synthesis ratio, which is a synthesis ratio between the first synthesized image and the reference image in each of a plurality of regions of the reference image, based on the first synthesis ratio.
13. A program that, when read and executed by a computer, causes the computer to execute each step of the image processing method described in claim 11.
14. A program that, when read and executed by a computer, causes the computer to execute each step of the image processing method described in claim 12.
15. A computer-readable storage medium storing the program according to claim 13 or 14.