Image processing device and image processing method

Abstract

An image processing device according to the present disclosure comprises an addition unit, a correction unit, and a generation unit. The addition unit generates a current addition image obtained by adding up a plurality of input images that are included in a current image group and are continuously captured. The correction unit corrects the plurality of input images according to the deviation between a previous addition image and the current addition image and adds up the same to generate a current corrected addition image. The generation unit generates a current composite image on the basis of a previous composite image and the current corrected addition image.

Description

Image processing device and image processing method

[0001] The present disclosure relates to an image processing device and an image processing method.

[0002] A technique for generating a composite image from a group of short-exposure images captured continuously with a fixed short exposure time is known. By aligning and combining the short-exposure images, a composite image with proper exposure and reduced camera and subject shake is obtained.

[0003] In this way, by using multiple short-exposure images, it is possible to generate a composite image with reduced blur. However, when an image is read out from an imaging device, readout noise occurs. While the impact of readout noise is small for a single image, the impact of readout noise becomes significant when multiple images are combined.

[0004] Therefore, a technology has been proposed for generating a group of short-exposure images using a SPAD (Single Photon Avalanche Diode) image sensor with low readout noise. For example, a system disclosed in Patent Document 1 divides a group of binary images into multiple blocks and generates an additive image for each block. The system estimates motion between the additive images using the generated additive images. The system corrects the group of binary images according to the estimated motion and generates a final image based on the corrected group of binary images.

[0005] US Patent Application Publication No. 2022 / 0067997

[0006] However, the above-described technique leaves room for improvement in terms of reducing the processing load. In the above-described technique, the system generates a final image using all images included in the binary image group. Therefore, as the number of images included in the binary image group increases, the amount of processing increases.

[0007] Therefore, the present disclosure proposes an image processing device and an image processing method that can reduce the processing load of image processing.

[0008] It should be noted that the above problem or object is merely one of multiple problems or objects that can be solved or achieved by multiple embodiments disclosed in this specification.

[0009] The image processing device of the present disclosure includes an adder, a correction unit, and a generator. The adder generates a current added image by adding together multiple input images included in a current image group that have been captured consecutively. The correction unit generates a current corrected added image by correcting and adding together the multiple input images according to a deviation between the previous added image and the current added image. The generator generates the current composite image based on the previous composite image and the current corrected added image.

[0010] 1 is a diagram illustrating an example of a conventional method for acquiring an image with proper exposure when shooting in a dark place; FIG. 2 is a diagram for explaining the influence of readout noise by a conventional method; FIG. 3 is a diagram illustrating an example of a method for generating a composite image by a comparison method; FIG. 4 is a diagram illustrating an example of performing reconstruction processing by a comparison method on a moving image; FIG. 5 is a diagram illustrating an example of reconstruction processing according to the proposed technology of the present disclosure; FIG. 6 is a diagram illustrating a detailed example of reconstruction processing according to the proposed technology of the present disclosure; FIG. 7 is a block diagram illustrating an example of a configuration of an electronic device according to an embodiment of the present disclosure; FIG. 8 is a diagram illustrating an example of reconstruction processing according to an embodiment of the present disclosure; FIG. 9 is a diagram illustrating a processing flow until an image processing device according to an embodiment of the present disclosure outputs a composite image; FIG. 10 is a flowchart illustrating an example of a flow of reconstruction processing according to an embodiment of the present disclosure; FIG. 11 is a flowchart illustrating an example of a flow of reconstruction processing according to an embodiment of the present disclosure; FIG. 12 is a diagram illustrating an example of reconstruction processing according to another embodiment of the present disclosure; FIG. 13 is a diagram illustrating another example of reconstruction processing according to another embodiment of the present disclosure;

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] In this specification and drawings, similar components of the embodiments may be distinguished by adding at least one different alphabet and / or number after the same reference numeral. However, if there is no need to particularly distinguish between the similar components, only the same reference numeral will be used.

[0013] In the following description, specific numerical values ​​may be used as examples, but these are merely examples and other numerical values ​​may be used.

[0014] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from each other. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects from each other.

[0015] <<1. Introduction>> <1-1. Problems> For example, when photographing in a dark place, attempting to capture an image with a proper exposure requires a long exposure time, resulting in camera or subject shake. To suppress this shake, a technique is known for generating an image with a proper exposure using multiple short-exposure images (hereinafter also referred to as a short-exposure image group) captured in succession with short exposure times.

[0016] Fig. 1 shows an example of a conventional method for acquiring properly exposed images when shooting in a dark place. As shown in Fig. 1, a group of short-exposure images (short-exposure RAW images in Fig. 1) captured with a constant short exposure time is acquired. The short-exposure images are underexposed but have little blur.

[0017] The short-exposure images are aligned with each other, and the aligned short-exposure images are combined into a single image to generate a combined image (a combined RAW image in FIG. 1).

[0018] In this way, by performing image reconstruction processing on a group of short-exposure images through alignment and synthesis, it is possible to reconstruct a composite image with proper exposure in which camera and subject shake is suppressed.

[0019] This technique is effective in improving the quality of images captured in dark places or scenes with a high dynamic range, and is widely used.

[0020] To further improve the blur suppression effect of this method, it is necessary to acquire more short-exposure images with shorter exposure times, in other words, to increase the number of frame divisions. However, acquiring more short-exposure images increases the impact of readout noise, which is a problem. In other words, there is a trade-off between the number of frame divisions and the impact of readout noise.

[0021] Fig. 2 is a diagram for explaining the influence of readout noise by a conventional method. As shown in the upper diagram of Fig. 2, for example, when normal photography, which obtains one image in one shot, is performed in a dark place, a long exposure time is used to ensure proper exposure.

[0022] The aforementioned readout noise occurs when the captured image is read out after shooting is completed. In normal shooting, the exposure time is long, but only one captured image is acquired, so readout noise occurs only once. Therefore, in this case, although camera or subject shake is likely to occur, the captured image is not easily affected by readout noise.

[0023] On the other hand, as shown in the lower diagram of Figure 2, in the conventional method of performing multiple short-exposure shots in succession, the exposure time is shorter than in normal shooting, but the number of times short-exposure images are acquired increases, which increases the number of times readout noise occurs each time a short-exposure image is acquired.

[0024] As a result, the composite image obtained using the conventional method is significantly affected by readout noise. In other words, while the conventional method can suppress camera and subject shake, the composite image is significantly affected by readout noise.

[0025] Therefore, to obtain a high-quality composite image using conventional methods, it is desirable to use an imaging device (image sensor) that can read out more short-exposure images with a shorter exposure time and lower readout noise. In other words, a high-quality composite image can be obtained by using an imaging device with a high frame rate and low readout noise.

[0026] A SPAD image sensor (hereinafter simply referred to as SPAD) is known as an imaging device that has a high frame rate and low readout noise. A SPAD is an image sensor that has a pixel structure that utilizes avalanche multiplication and counts the number of incident photons as a digital value.

[0027] Here, a CMOS image sensor (hereinafter simply referred to as a CIS) acquires an image by AD converting accumulated electric charges. AD conversion generates readout noise. Furthermore, the readout time is determined by the AD conversion. In other words, the AD conversion determines the frame rate of the CIS.

[0028] On the other hand, as mentioned above, SPAD counts the number of incident photons as a digital value. Therefore, SPAD does not require AD conversion during readout, and is less likely to generate readout noise. Furthermore, because AD conversion is not required, the frame rate is not limited by AD conversion, making high-speed imaging possible.

[0029] As described above, SPAD has the advantages of a higher frame rate and lower read noise than CIS, which makes it more suitable for conventional methods of generating a composite image from a group of short-exposure images.

[0030] As a method for generating a composite image using SPAD, there is a method for performing alignment using an added image of a group of short-exposure images (hereinafter also referred to as a comparison method).

[0031] Fig. 3 is a diagram showing an example of a method for generating a composite image using a comparison technique. Fig. 3 shows an example in which an image processing device (not shown) generates a composite image 40a from multiple input images 10 (first to ninth input images 10_1 to 10_9 in the example of Fig. 3) that have been captured consecutively. Note that the numerical values ​​given here (e.g., the number of input images 10 and the number of blocks 20a described below) are merely examples and are not limited to the numerical values ​​given here.

[0032] A plurality of input images 10 are divided into a plurality of pixel groups 20a (hereinafter also simply referred to as blocks 20a) by an image processing device. In the example of Fig. 3, the first to ninth input images 10_1 to 10_9 are divided into three blocks 20a_1 to 20a_3.

[0033] The first block 20a_1 includes the first to third input images 10_1 to 10_3. The second block 20a_2 includes the fourth to sixth input images 10_4 to 10_6. The third block 20a_3 includes the seventh to ninth input images 10_7 to 10_9. In other words, one block 20a includes three input images 10.

[0034] The image processing device adds the input images 10 included in each divided block 20a to generate an additive image 30a. For example, the image processing device adds the first to third input images 10_1 to 10_3 included in the first block 20a_1 to generate a first additive image 30a_1. In a similar manner, the image processing device generates a second additive image 30a_2 and a third additive image 30a_3.

[0035] Next, the image processing device estimates the motion between the plurality of added images 30 a. For example, the image processing device estimates the motion of the subject between a reference added image 30 a (e.g., the second added image 30 a_2) and other added images 30 a (e.g., the first and third added images 30 a_1 and 30 a_3).

[0036] The image processing device calculates a correction value for the other input image 10 relative to the reference input image 10 from the estimated movement of the image (more specifically, the object). This correction value is the amount of movement (warping) of each pixel value of the input image 10. The image processing device moves the pixel values ​​of the other input image 10 by the correction value, thereby aligning the position of the object included in the other input image 10 with the reference input image 10.

[0037] 3, for example, the image processing device uses the fifth input image 10_5 as a reference and calculates correction values ​​for the remaining input images 10. For example, the image processing device determines the motion (motion estimated value or movement amount) between the second added image 30a_2 serving as a reference and the first added image 30a_1 as the motion (motion estimated value) between the fifth input image 10_5 serving as a reference and the second input image 10_2. The image processing device calculates the correction amount for the second input image 10_2 from this movement amount.

[0038] Similarly, for example, the image processing device determines the motion (motion estimate) between the second added image 30 a_2 and the third added image 30 a_3 as the motion (motion estimate) between the fifth input image 10_5 and the eighth input image 10_8 as the reference, and calculates the correction amount for the eighth input image 10_8 from this amount of movement.

[0039] The image processing device calculates the correction amounts for the first, third, and fourth input images 10_1, 10_3, and 10_4 by linear interpolation of the correction amount for the second input image 10_2, and calculates the correction amounts for the sixth, seventh, and ninth input images 10_6, 10_7, and 10_9 by linear interpolation of the correction amount for the eighth input image 10_8.

[0040] The image processing device corrects each input image 10 based on the calculated correction amount to generate a corrected input image. The image processing device divides the generated corrected input image into a plurality of blocks 21a (hereinafter also referred to as correction blocks 21a).

[0041] For example, the first correction block 21a_1 includes the first to third corrected input images, the second correction block 21a_2 includes the fourth to sixth corrected input images, and the third correction block 21a_3 includes the seventh to ninth corrected input images.

[0042] The image processing device adds the corrected input images included in each correction block 21a to generate a corrected added image 31a. In the example of Fig. 3, the image processing device generates first to third corrected added images 31a_1 to 31a_3.

[0043] The image processing device combines the first to third corrected added images 31a_1 to 31a_3 to generate a combined image 40a.

[0044] In this way, the image processing device generates a composite image 40a from a plurality of continuously captured input images 10. This processing described above is also referred to as image reconstruction processing.

[0045] The reconstruction process using the comparative method described above has room for improvement in terms of the amount of processing required. In the reconstruction process using the comparative method, multiple input images 10 are divided into multiple blocks 20a, and processing such as addition, motion estimation, and correction is performed on each block 20a.

[0046] The image processing device must perform these processes every time it generates a composite image 40a, and the amount of processing increases as the number of composite images 40a to be generated increases.

[0047] In particular, in the case of moving images, when each image included in the moving image (hereinafter also referred to as a frame image) is generated by reconstruction processing using a comparison method, the processing load becomes large.

[0048] The technique of performing reconstruction processing on continuously captured input images 10 can be applied not only to still images but also to generating moving images. In particular, when the imaging device that captures the input images 10 is a SPAD, the image processing device can generate moving images that take advantage of the features of the SPAD (e.g., high frame rate and low read noise). For example, the image processing device can generate moving images captured by super slow motion shooting in an extremely dark environment.

[0049] In this way, by applying the reconstruction process to moving images, it is expected that moving images that could not be captured conventionally can be captured.

[0050] However, as mentioned above, in the reconstruction process using the comparison method, processes such as addition, motion estimation, and correction are performed each time a frame image is generated. Therefore, the algorithm means that the image processing device processes the same image multiple times, which increases the processing load (e.g., the amount of calculation and processing time). This point will be explained using FIG. 4.

[0051] 4 is a diagram showing an example of executing a reconstruction process using a comparison method on a moving image, in which one composite image 40a (frame image) is reconstructed from 400 input images 10.

[0052] For example, the image processing device performs reconstruction processing using a comparison method on the first to four hundredth input images 10_1 to 10_400, and generates a first composite image 40a_1 (first frame image).

[0053] Specifically, the image processing device divides the first to 400th input images 10_1 to 10_400 into a plurality of blocks 20a, and calculates an added image 30a for each block 20a. The image processing device estimates the motion between the added images 30a, and calculates the amount of correction for each of the first to 400th input images 10_1 to 10_400.

[0054] The image processing device corrects the first to four hundredth input images 10_1 to 10_400 based on the calculated correction amounts. The image processing device generates a corrected added image 31a from the first to four hundredth corrected input images, and generates a first composite image 40a_1.

[0055] Next, the image processing device performs reconstruction processing using a comparison method on the second to 401st input images 10_2 to 10_401 to generate a second composite image 40a_2 (second frame image).

[0056] Specifically, the image processing device divides the second to 401st input images 10_2 to 10_401 into a plurality of blocks 20a, and calculates an added image 30a for each block 20a. The image processing device estimates the motion between the added images 30a, and calculates the amount of correction for each of the second to 401st input images 10_2 to 10_401.

[0057] The image processing device corrects the second to 401st input images 10_2 to 10_401 based on the calculated correction amounts. The image processing device generates a corrected added image 31a from the second to 401st corrected input images, and generates a second composite image 40a_2.

[0058] Next, the image processing device performs reconstruction processing using a comparison method on the third to 402nd input images 10_3 to 10_402 to generate a third composite image 40a_3 (third frame image).

[0059] Specifically, the image processing device divides the third to 402nd input images 10_3 to 10_402 into a plurality of blocks 20a, and calculates an added image 30a for each block 20a. The image processing device estimates the motion between the added images 30a, and calculates the amount of correction for each of the third to 402nd input images 10_3 to 10_402.

[0060] The image processing device corrects the third to 402nd input images 10_3 to 10_402 based on the calculated correction amounts. The image processing device generates a corrected added image 31a from the third to 402nd corrected input images, and generates a third composite image 40a_3.

[0061] By sequentially generating the composite images 40a, the image processing device can generate a moving image with correct exposure and reduced blur, even in a dark place. However, each time the composite image 40a is generated, the image processing device processes the same image multiple times.

[0062] For example, when generating the first and second composite images 40a_1 and 40a_2, respectively, the image processing device processes the same second to 400th input images 10_2 to 10_400.

[0063] Therefore, if the reconstruction process using the comparison method is applied as is to the generation of a plurality of composite images 40a such as a moving image, there is a problem that the processing load on the image processing device increases.

[0064] Therefore, the present disclosure proposes reconstruction processing (an example of image processing) that can reduce the processing load.

[0065] 5 is a diagram showing an example of a reconstruction process according to the proposed technique of the present disclosure. The reconstruction process shown in FIG. 5 is executed by, for example, an image processing device (not shown).

[0066] First, the image processing device generates a first composite image 40_1 (first composite image) using a zeroth block 20_0 (an example of the zeroth image group) and a first block 20_1 (an example of the first image group). In the example of Fig. 5, each block 20 includes three input images 10, but the number of input images 10 included in each block 20 is not limited to three as long as it is plural.

[0067] The image processing device generates a first composite image 40_1 by performing a reconstruction process using, for example, the above-mentioned comparison method.

[0068] Next, the image processing device generates a second composite image 40_2 by executing the reconstruction process of the proposed technology using the second block 20_2 and the result of the previous reconstruction process (the first composite image 40_1 in the example of FIG. 5 ). Hereinafter, the reconstruction process of the proposed technology will be referred to as the reconstruction process by this technique or simply as the reconstruction process.

[0069] The image processing device generates a third composite image 40_3 by performing the reconstruction process of the proposed technology using the third block 20_3 and the result of the previous reconstruction process (in the example of Figure 5, the second composite image 40_2).

[0070] In this way, in the reconstruction process of this method, the image processing device uses the results of the previous reconstruction process to generate the current composite image 40. This allows the image processing device to reduce the amount of processing in the current reconstruction process.

[0071] Although the number of input images 10 included in each block 20 is assumed to be three here, the number of input images 10 included in each block 20 is not limited to three. The number of input images 10 included in each block 20 may be two or four or more as long as it is more than one.

[0072] 6 is a diagram showing a detailed example of the reconstruction processing according to the proposed technique of the present disclosure. In FIG. 6, an image processing device (not shown) according to the proposed technique of the present disclosure generates an n-th composite image 40_n (n is an integer equal to or greater than 1) from n-th to n+4-th input images 10_n to 10_n+4. In this case, the image processing device executes the current (n-th) reconstruction processing described here using the result of the previous (n-1-th) reconstruction processing.

[0073] The image processing device divides the (n-1)th to (n+4)th input images 10_n-1 to 10_n+4, which have been captured consecutively, into two image groups (the (n-1)th and n-th blocks 20_n-1 and 20_n). In the example of Fig. 6, the image processing device generates the (n-1)th block 20_n-1 including the (n-1)th to (n+3)th input images 10_n-1 to 10_n+3, and the n-th block 20_n including the (n-1)th to (n+4)th input images 10_n to 10_n+4.

[0074] In this way, the image processing device generates two groups of input images (for example, blocks 20) from a plurality of input images 10 captured successively. Parts of the input images 10 included in different blocks 20 overlap. For example, in the (n-1)th and (n)th blocks 20_n-1 and 20_n, the (n)th to (n+3)th input images 10_n to 10_n+3 overlap in each block 20.

[0075] The image processing device adds the nth to (n+4th)th input images 10_n to 10_n+4 included in the nth block 20_n to generate the nth added image 30_n.

[0076] The image processing device estimates the motion between the nth added image 30_n and the (n-1)th added image 30_n-1 generated in the previous reconstruction process. This (n-1)th added image 30_n-1 is an image obtained by adding the (n-1)th to (n+3)th input images 10_n-1 to 10_n+3 included in the (n-1)th block 20_n-1 during the (n-1)th reconstruction process.

[0077] The image processing device corrects the nth to n+4th input images 10_n to 10_n+4 according to the results of motion estimation, and aligns the positions of the input images 10. The nth block including the aligned input images (hereinafter also referred to as the corrected input images) is also referred to as the nth corrected block 21_n.

[0078] The image processing device adds the corrected input images included in the n-th correction block 21_n to generate a corrected added image 31_n.

[0079] The image processing device generates the n-th composite image 40_n using the corrected added image 31_n and the (n-1)th composite image 40_n-1 generated in the previous (n-1)th) reconstruction process.

[0080] For example, the image processing device estimates the motion between the corrected added image 31_n and the (n-1)th composite image 40_n-1, and corrects the (n-1)th composite image 40_n-1 in accordance with the result of the motion estimation.

[0081] Specifically, the image processing device moves (warps) each pixel value of the (n-1)th composite image 40_n-1 to match the position of the subject in the corrected added image 31_n according to the results of motion estimation, and generates the (n-1)th corrected composite image 41_n-1.

[0082] The image processing device combines the corrected added image 31_n with the (n-1)th corrected combined image 41_n-1 to generate the nth combined image 40_n.

[0083] In this way, the image processing device generates the nth composite image 40_n using the results of the previous (n-1th) reconstruction process (for example, the (n-1)th added image 30_n-1 and the (n-1)th composite image 40_n-1). By using the results of the previous reconstruction process, the image processing device can reduce the number of input images to be added and corrected for the input image 10, thereby reducing the processing load.

[0084] Furthermore, the images required for the nth reconstruction process are the nth to n+4th input images 10_n to 10_n+4 contained in the nth block 20_n, the n-1th added image 30_n-1, and the n-1th composite image 40_n-1.

[0085] This number of images is smaller than the nth to n+4th input images 10_n to 10_n+4 contained in the nth block 20_n and the nth to n+3rd input images 10_n-1 to 10_n+3 contained in the n-1th block 20_n-1.

[0086] That is, when the nth composite image 40_n is generated from the n-1th and nth blocks 20_n-1 and 20_n each time without using the result of the n-1th reconstruction process, the number of images that the image processing device must hold is 10. On the other hand, when the result of the n-1th reconstruction process is used, the number of images that the image processing device must hold is 7.

[0087] In this way, by executing the n-th reconstruction process using the result of the (n-1)-th reconstruction process, the image processing device can reduce the number of images held during the reconstruction process. In other words, the image processing device can reduce the frame memory required for processing.

[0088] As described above, the image processing device according to the proposed technique of the present disclosure can reduce the amount of calculation and processing time of the reconstruction process, and can also reduce the frame memory required for the process.

[0089] Although the number of input images 10 included in each block 20 is assumed to be five here, the number of input images 10 included in each block 20 is not limited to five. The number of input images 10 included in each block 20 may be more than one, and may be four or less, or six or more.

[0090] It is also assumed that some of the input images 10 included in each block 20 overlap. In the example of FIG. 6 , four of the five input images 10 included in each block 20 overlap. The number of overlapping input images 10 is not limited to four. It is sufficient that at least one input image 10 overlaps in each block 20. For example, of the five input images 10 included in each block 20, the number of overlapping input images 10 may be three or less.

[0091] 6, one of the input images 10 included in each block 20 does not overlap. The image processing device generates one block 20 from five consecutive input images 10. The image processing device sequentially generates consecutive blocks 20 by shifting the consecutive input images 10 one by one.

[0092] The number of input images 10 to be shifted when sequentially generating consecutive blocks 20 (hereinafter also referred to as the amount of shift) is not limited to 1 as described above and may be 2 or more. For example, when generating one block 20 from five input images 10, if the amount of shift is 2, the image processing device generates one block 20 from the n-2th to n+2th input images 10_n-2 to 10_n+2, and generates one block 20 from the nth to n+4th input images 10_n to 10_n+4.

[0093] 7 is a block diagram showing a configuration example of an electronic device 300 according to an embodiment of the present disclosure. The electronic device 300 is, for example, a camera. The electronic device 300 in FIG. 7 includes an imaging device 200 and an image processing device 100.

[0094] (Image capture device 200) The image capture device 200 captures an image of the surroundings of the electronic device 300. The image capture device 200 continuously captures a plurality of images and generates a plurality of input images 10. The image capture device 200 outputs the generated input images 10 to the image processing device 100.

[0095] The imaging device 200 may be of any type as long as it can take multiple short-exposure images consecutively. For example, the imaging device 200 may be a CMOS image sensor or a SPAD image sensor. Alternatively, the imaging device 200 may be an image sensor other than a CMOS image sensor or a SPAD image sensor.

[0096] However, when performing continuous imaging at a high frame rate and low read noise, the imaging device 200 is preferably a SPAD image sensor.

[0097] (Image Processing Device 100) The image processing device 100 is an information processing device that processes input images 10 (image information) captured by the imaging device 200. The image processing device 100 performs the reconstruction process of the present method on multiple input images 10 captured consecutively, and generates a composite image 40.

[0098] The image processing device 100 shown in FIG. 7 includes a storage unit 110, a display unit 120, and a control unit 130.

[0099] (Storage Unit 110) The storage unit 110 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 110 stores, for example, the input image 10 and the composite image 40. The storage unit 110 may also store images generated during the reconstruction process, such as the corrected added image 31 and the corrected composite image 41.

[0100] (Display Unit 120) The display unit 120 is a mechanism for outputting various types of information, and is, for example, a liquid crystal display. For example, the display unit 120 displays the input image 10 captured by the imaging device 200, or the composite image 40 composited by the image processing device 100. The display unit 120 may also function as a processing unit for accepting various operations from a user or the like who uses the image processing device 100. For example, the display unit 120 may accept input of various types of information via key operations, a touch panel, or the like.

[0101] Although the image processing device 100 is described here as including the display unit 120, the image processing device 100 may not necessarily include the display unit 120. For example, the image processing device 100 may display an image on an external display device (not shown). In this case, the display device may be located in the same place as the image processing device 100 or in a different place. If the display device is located in a different place, the image processing device 100 and the display device may be connected to each other via a network (for example, the Internet).

[0102] (Control Unit 130) The control unit 130 is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), or the like executing a program stored inside the image processing device 100 using a RAM or the like as a work area. The control unit 130 is a controller, and may be realized by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0103] 7, the control unit 130 has an acquisition unit 131, an addition unit 132, a correction unit 133, and a synthesis unit 134, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 7, and may be any other configuration as long as it is capable of performing the information processing (reconstruction processing) described below.

[0104] (Acquisition Unit 131) The acquisition unit 131 acquires the input image 10 from the imaging device 200. The acquisition unit 131 acquires from the storage unit 110 the added image 30 and the composite image 40 generated in the previous reconstruction process.

[0105] The acquisition unit 131 outputs the input image 10 to the addition unit 132 and the correction unit 133. The acquisition unit 131 outputs the previous added image 30 to the correction unit 133. The acquisition unit 131 outputs the previous composite image 40 to the composition unit .

[0106] (Addition unit 132) The addition unit 132 selects an input image 10 to be used in the current reconstruction process from the input images 10 input from the acquisition unit 131, and generates a block 20 including the selected input image 10. The addition unit 132 adds the input images 10 included in the block 20 to generate an added image 30. The addition unit 132 outputs the generated added image 30 to the correction unit 133 and the storage unit 110.

[0107] When the addition unit 132 performs the reconstruction process for the first time, it generates two blocks, ie, the 0th and 1st blocks 20_0 and 20_1, and two summed images, ie, the 0th and 1st summed images 30_0 and 30_1. On the other hand, when the addition unit 132 performs the reconstruction process for the second time or later, it generates one block 20 and one summed image 30.

[0108] Although the adding unit 132 generates the block 20 here, the component that generates the block 20 is not limited to the adding unit 132. For example, the acquiring unit 131 may generate the block 20.

[0109] In this case, the acquisition unit 131 generates the block 20 by, for example, selecting an input image 10 to be used in the reconstruction process from the input images 10 acquired from the imaging device 200. The acquisition unit 131 outputs the input image 10 included in the generated block 20 to the addition unit 132. At this time, the acquisition unit 131 may be configured to output the input image 10 to the addition unit 132 together with information related to the generated block 20.

[0110] (Correction Unit 133) The correction unit 133 corrects (aligns) the input image 10 included in the block 20 generated by the addition unit 132, and generates a corrected input image.

[0111] The correction unit 133 performs motion estimation by comparing the current added image 30 generated by the addition unit 132 with the previous added image 30 acquired from the acquisition unit 131. The correction unit 133 estimates the amount of motion of each pixel value between the current added image 30 and the previous added image 30 (motion estimation value, for example, an example of a deviation between images).

[0112] The correction unit 133 calculates a correction value for each input image 10 included in the block 20 generated in the current reconstruction process. The correction unit 133 calculates the correction value for each input image 10 from, for example, the estimated amount of motion of each pixel value.

[0113] The correction unit 133 determines that the amount of movement of each pixel value between the reference input image 10 in the current block 20 and the reference input image 10 in the previous block 20 is the amount of movement of each pixel value between the current added image 30 and the previous added image 30.

[0114] If the current block 20 contains the same input image 10 as the reference input image 10 in the previous block 20, the correction unit 133 determines that the estimated movement amount of each pixel value is the movement amount of each pixel value between the input image 10 and the reference input image 10 in the current block 20, i.e., the correction amount.

[0115] The correction unit 133 calculates the amount of correction for the remaining input image 10 in the current block 20 using, for example, linear interpolation.

[0116] The correction unit 133 corrects the input image 10 using the calculated correction amount to generate a corrected input image. The correction unit 133 shifts (warps) each pixel of the input image 10 other than the reference input image 10 by the correction amount.

[0117] As a result, the correction unit 133 generates a corrected input image that is aligned with the reference input image 10. Note that the reference input image 10 can be generated with a correction amount of zero, and a corresponding corrected input image can be generated.

[0118] The correction unit 133 adds the corrected input images to generate a corrected added image 31. The correction unit 133 outputs the corrected added image 31 to the composition unit 134. Note that the correction unit 133 may output the corrected input images, and the composition unit 134 may generate the corrected added image 31.

[0119] Here, if the previous additive image 30 cannot be obtained, i.e., if the reconstruction process is performed for the first time, the correction unit 133 uses the additive image 30 of the input image 10 in the 0th block 20_0, i.e., the 0th additive image 30_0, instead of the previous additive image 30.

[0120] The correction unit 133 performs motion estimation between the summed image 30 of the input image 10 in the first block 20_1, that is, the first summed image 30_1 and the 0th summed image 30_0.

[0121] The correction unit 133 corrects the input image 10 in the first block 20_1 (moves each pixel value) based on motion estimation, thereby aligning the remaining input image 10 with the reference input image 10 and generating a corrected input image.

[0122] In addition, the correction unit 133 may correct the input image 10 in the 0th block 20_0 (moving each pixel value) based on motion estimation to generate a corrected input image for use in processing in the subsequent synthesis unit 134.

[0123] (Synthesizing Unit 134) The synthesizing unit 134 adds the corrected input images generated by the correction unit 133 to generate a corrected added image 31. The synthesizing unit 134 performs motion estimation between the corrected added image 31 and the previous synthesized image 40.

[0124] The synthesis unit 134 corrects the previous synthesized image 40 in accordance with the result of the motion estimation, and generates a corrected synthesized image 41. This corrected synthesized image 41 is an image that has been aligned with the corrected added image 31.

[0125] The composition unit 134 combines the corrected added image 31 and the corrected composite image 41 to generate the current composite image 40 .

[0126] The composition unit 134 may store the generated composite image 40 in, for example, the storage unit 110. Alternatively, if the generated composite image 40 satisfies a predetermined condition, the composition unit 134 presents the composite image 40 to the user by outputting it to the display unit 120. If the predetermined condition is satisfied, the composition unit 134 may store the composite image 40 in the storage unit 110. Examples of the predetermined condition include when the signal-to-noise ratio of the composite image 40 exceeds a predetermined threshold, or when the number of iterations of the reconstruction process exceeds a predetermined threshold.

[0127] When the reconstruction process is performed for the first time, the previous composite image 40 does not exist. Therefore, the composition unit 134 adds the corrected input image generated by the correction unit 133, which is obtained by correcting the input image 10 included in the 0th block 20_0, to generate the 0th corrected added image 31_0. The composition unit 134 also adds the corrected input image generated by correcting the input image 10 included in the first block 20_1, to generate the first corrected added image 31_1.

[0128] The composition unit 134 performs motion estimation between the 0th corrected and added image 31_0 and the first corrected and added image 31_1. Based on the result of the motion estimation, the composition unit 134 corrects the 0th corrected and added image 31_0 and aligns it with the first corrected and added image 31_1.

[0129] The combining unit 134 combines the first corrected and added image 31_1 with the aligned 0th corrected and added image 31_0 to generate a first combined image 40_1.

[0130] <2-2. Processing Example> Next, a detailed example of the processing executed by each unit of the image processing device 100 will be described. Figures 8 and 9 are diagrams showing an example of reconstruction processing according to an embodiment of the present disclosure. Figures 8 and 9 show an example of the nth (nth) reconstruction processing.

[0131] 8 and 9, the number of input images 10 included in a block 20 is "3," and the amount of deviation between adjacent blocks 20 (the number of non-overlapping input images 10) is "1." Note that these numerical values ​​are merely examples, and the present invention is not limited to these numerical values.

[0132] First, the adder 132 of the image processing device 100 generates an n-th block 20_n. The n-th block 20_n includes the n-th, (n+1)th, and (n+2)th input images 10_n, 10_n+1, and 10_n+2.

[0133] The adder 132 adds the nth, n+1th, and n+2nd input images 10_n, 10_n+1, and 10_n+2 of the nth block 20_n to generate the nth added image 30_n.

[0134] The correction unit 133 performs motion estimation using the nth added image 30_n and the n-1th added image 30_n-1 generated in the previous (n-1th) reconstruction process, and calculates the amount of motion of each pixel value.

[0135] The correction unit 133 calculates the amount of motion of the nth and (n-1)th added images 30_n and 30_n-1 for each pixel using the nth added image 30_n as a reference. In Fig. 8, the calculated amount of motion for each pixel is shown as images 51_n and 51_n-1. Image 51_n is an image that shows the amount of motion of the nth added image 30_n using the nth added image 30_n as a reference. Image 51_n-1 is an image that shows the amount of motion of the (n-1)th added image 30_n-1 using the nth added image 30_n as a reference.

[0136] Next, the correction unit 133 estimates the amount of motion of the remaining input images 10 (in the example of Figure 8, the nth and n+2th input images 10_n and 10_n+2) relative to the reference input image 10 (in the example of Figure 8, the nth+1th input image 10_n+1) among the input images 10 included in the nth block 20_n.

[0137] For example, the correction unit 133 assumes that the amount of motion of the (n-1)th added image 30_n-1 relative to the nth added image 30_n is the amount of motion of the input image 10 (hereinafter also referred to as the reference input image 10_n-1R) that serves as the reference for the (n-1)th block 20_n-1 relative to the nth block 20_n.

[0138] For example, among the input images 10 included in the nth block 20_n, the central input image 10 (in the example of FIG. 8, the (n+1)th input image 10_n+1) is assumed to be the reference input image 10_nR. In this case, the reference input image 10_n-1R ​​of the (n-1)th block 20_n-1 becomes the nth input image 10_n.

[0139] The correction unit 133 determines the amount of motion of the (n-1)th added image 30_n-1 relative to the nth added image 30_n as the amount of motion of the reference input image 10_n-1R ​​of the (n-1)th block 20_n-1 relative to the reference input image 10_nR of the nth block 20_n. In other words, the correction unit 133 estimates that the amount of motion of the nth input image 10_n relative to the (n+1)th input image 10_n+1 is the amount of motion of the (n-1)th added image 30_n-1 relative to the nth added image 30_n.

[0140] The amount of motion of the remaining input image 10 included in the nth block 20_n (in the example of FIG. 8 , the (n+2)th input image 10_n+2) is estimated based on the amount of motion of the nth input image 10_n relative to the (n+1)th input image 10_n+1. The correction unit 133 estimates the amount of motion of the (n+2)th input image 10_n+2, for example, by linear interpolation or the like, based on the amount of motion of the nth input image 10_n relative to the (n+1)th input image 10_n+1.

[0141] The correction unit 133 calculates the amount of motion for each pixel of the nth to n+2nd input images 10_n to 10_n+2 using the (n+1)th input image 10_n+1 as a reference. In Fig. 8, the calculated amount of motion for each pixel is shown as images 52_n to 52_n+2.

[0142] Image 52_n is an image that shows the amount of motion of the nth input image 10_n using the (n+1)th input image 10_n+1 as a reference. Image 52_n+1 is an image that shows the amount of motion of the (n+1)th input image 10_n+1 using the (n+1)th input image 10_n+1 as a reference. Image 52_n+2 is an image that shows the amount of motion of the (n+2)th input image 10_n+2 using the (n+1)th input image 10_n+1 as a reference.

[0143] The correction unit 133 corrects the input image 10 included in the nth block 20_n using the estimated amount of motion as the correction amount. The correction unit 133 generates the nth to n+2th corrected input images by shifting (warping) each pixel value of the nth to n+2nd input images 10_n to 10_n+2 according to the amount of motion. The nth to n+2th corrected input images are images that have been aligned with the reference input image 10_nR.

[0144] Here, the reference input image 10_nR of the n-th block 20_n is not limited to the central input image 10 among the multiple input images 10 included in the n-th block 20_n. The reference input image 10_nR may be one of the input images 10 included in the n-th block 20_n. For example, the reference input image 10_nR may be the oldest (or newest) input image 10 among the multiple input images 10 included in the n-th block 20_n.

[0145] For example, if the reference input image 10_nR is the oldest nth input image 10_n in the nth block 20_n, the reference input image 10_n-1R ​​of the n-1th block 20_n-1 will be the oldest n-1th input image 10_n-1 in the n-1th block.

[0146] The correction unit 133 determines the amount of motion of the (n-1)th added image 30_n-1 relative to the nth added image 30_n as the amount of motion of the reference input image 10_n-1R ​​of the (n-1)th block 20_n-1 relative to the reference input image 10_nR of the nth block 20_n. In other words, the correction unit 133 estimates that the amount of motion of the (n-1)th input image 10_n-1 relative to the nth input image 10_n is the amount of motion of the (n-1)th added image 30_n-1 relative to the nth added image 30_n.

[0147] The amount of motion of the remaining input images 10 included in the nth block 20_n (for example, the n+1th and n+2nd input images 10_n+1 and 10_n+2) is estimated based on the amount of motion of the n-1th input image 10_n-1 relative to the nth input image 10_n.

[0148] The correction unit 133 estimates the amount of motion of the n+1st and n+2nd input images 10_n+1 and 10_n+2, for example, by linear interpolation or the like, based on the amount of motion of the n-1st input image 10_n-1 relative to the nth input image 10_n.

[0149] The synthesis unit 134 adds the n-th to (n+2)-th corrected input images to generate the n-th corrected added image 31_n.

[0150] Next, as shown in Figure 9, the synthesis unit 134 performs motion estimation using the nth added image 30_n and the n-1th added image 30_n-1 generated in the previous (n-1th) reconstruction process, and calculates the amount of motion of each pixel value.

[0151] The composition unit 134 calculates the amount of motion of the (n-1)th composite image 40_n-1 for each pixel using the nth corrected and added image 31_n as a reference. In Figure 9, the calculated amount of motion for each pixel is shown as image 53_n-1. In other words, image 53_n-1 is an image that indicates the amount of motion of the (n-1)th composite image 40_n-1 using the nth corrected and added image 31_n as a reference.

[0152] The composition unit 134 corrects the input image 10 included in the n-th block 20_n using the estimated amount of motion as a correction amount. The composition unit 134 generates the n-1th corrected composite image 41_n-1 by shifting (warping) each pixel value of the (n-1)th composite image 40_n-1 according to the amount of motion. The (n-1)th corrected composite image 41_n-1 is an image aligned with the corrected added image 31_nR.

[0153] The composition unit 134 combines the n-th corrected added image 31_n and the (n-1)-th corrected composite image 41_n-1 to generate the n-th composite image 40_n.

[0154] 10 is a diagram illustrating a processing flow until the image processing device 100 according to an embodiment of the present disclosure outputs the composite image 40. Here, the image processing device 100 outputs the composite image 40 to the display unit 120 when the S / N ratio of the composite image 40 exceeds a predetermined threshold.

[0155] 10, the image processing device 100 generates a first composite image 40_1 using the 0th and 1st blocks 20_0 and 20_1. This first composite image 40_1 is not output because the S / N ratio does not exceed a predetermined threshold.

[0156] Next, the image processing device 100 executes the current reconstruction process using the second block 20_2 and the first composite image 40_1, which is the result of the previous reconstruction process, or the first added image 30_1, to generate a second composite image 40_2. This second composite image 40_2 is not output because its S / N ratio does not exceed a predetermined threshold.

[0157] The image processing device 100 generates third to eighth composite images 40_3 to 40_8 in the same manner.

[0158] The image processing device 100 executes the current reconstruction process using the ninth block 20_9 and the eighth composite image 40_8 that is the result of the previous reconstruction process, or the eighth added image 30_8, to generate the ninth composite image 40_9. Since the S / N ratio of this ninth composite image 40_9 exceeds a predetermined threshold, it is output to the display unit 120 and presented to the user.

[0159] The image processing device 100 executes the current reconstruction process using the tenth block 20_10 and the ninth composite image 40_9, which is the result of the previous reconstruction process, or the ninth added image 30_9, to generate the tenth composite image 40_10. Since the S / N ratio of this tenth composite image 40_10 exceeds a predetermined threshold, it is output to the display unit 120 and presented to the user.

[0160] Thereafter, the image processing device 100 generates a composite image 40 in the same manner, and outputs to the display unit 120 the composite image 40 whose S / N ratio exceeds the predetermined threshold.

[0161] In addition, if the S / N ratio of the composite image 40 exceeds a predetermined threshold even once, the image processing device 100 may thereafter output the generated composite image 40 to the display unit 120 regardless of the S / N ratio of the composite image 40.

[0162] Furthermore, even if the signal-to-noise ratio of the composite image 40 does not exceed a predetermined threshold, the composite image 40 generated by the image processing device 100 may be output to the display unit 120 when the number of repetitions of the reconstruction process exceeds a predetermined threshold (predetermined number of times).

[0163] Furthermore, the output destination of the composite image 40 is not limited to the display unit 120. For example, the image processing device 100 may output the composite image 40 to the storage unit 110.

[0164] Furthermore, image processing device 100 may change the threshold value for determining whether or not to output depending on the output destination. For example, the predetermined threshold value for determining whether or not to output when the output destination is display unit 120 may be greater than the predetermined threshold value for determining whether or not to output when the output destination is storage unit 110.

[0165] Fig. 11 is a flowchart showing an example of the flow of a reconstruction process according to an embodiment of the present disclosure. The reconstruction process shown in Fig. 11 is executed by the image processing device 100 when the imaging device 200 starts capturing images and generates the composite image 40 for the first time. Alternatively, this reconstruction process may be executed in accordance with an instruction from a user. This reconstruction process is executed once prior to the reconstruction process described with reference to Fig. 12.

[0166] 11 , the image processing device 100 acquires input images 10 from the imaging device 200 (step S101). The image processing device 100 determines whether or not a composite image 40 can be generated from the acquired input images 10 (step S102). For example, when the image processing device 100 acquires the number of input images 10 required to generate the composite image 40, it determines that the composite image 40 can be generated.

[0167] If the composite image 40 cannot be generated from the acquired input image 10 (step S102; No), the image processing device 100 returns to step S101 and acquires the input image 10.

[0168] On the other hand, if the composite image 40 can be generated from the acquired input image 10 (step S102; Yes), the image processing device 100 generates the 0th image group (0th block 20_0) (step S103).

[0169] Next, the image processing device 100 adds the input image 10 included in the 0th block 20_0 to generate the 0th added image (0th added image 30_0) (step S104).

[0170] The image processing device 100 generates a first image group (first block 20_1) (step S105). Next, the image processing device 100 adds the input images 10 included in the first block 20_1 to generate a first added image (first added image 30_1) (step S106).

[0171] The image processing device 100 performs motion estimation between the first added image 30_1 and the 0th added image 30_0 (step S107). The image processing device 100 calculates correction values ​​for each of the input images 10 included in the 0th and first blocks 20_0 and 20_1 according to the motion estimation results (step S108).

[0172] The image processing device 100 corrects the input images 10 included in the 0th and first blocks 20_0 and 20_1, respectively, and aligns the positions of the input images 10 (step S109).

[0173] The image processing device 100 adds the corrected input image 10 (corrected input image) for each block 20 to generate a corrected added image 31 (0th and first corrected added images 31_0 and 31_1) (step S110).

[0174] The image processing apparatus 100 combines the 0th and first corrected added images 31_0 and 31_1 to generate a combined image 40 (first combined image 40_1) (step S111).

[0175] The image processing device 100 determines whether the S / N ratio of the generated composite image 40 exceeds a threshold value (predetermined threshold value) (step S112).

[0176] If the S / N ratio of the composite image 40 does not exceed the threshold, i.e., if the S / N ratio is equal to or less than the threshold (step S112; No), the image processing device 100 ends the process. On the other hand, if the S / N ratio of the composite image 40 exceeds the threshold (step S112; Yes), the image processing device 100 outputs the composite image 40 to, for example, the display unit 120 (step S113), and ends the process.

[0177] Fig. 12 is a flowchart showing an example of the flow of reconstruction processing according to an embodiment of the present disclosure. The reconstruction processing shown in Fig. 12 is executed by the image processing device 100 after the reconstruction processing shown in Fig. 11. This reconstruction processing is repeatedly executed by the image processing device 100 while the input image 10 is being acquired from the imaging device 200. Alternatively, this reconstruction processing may be repeatedly executed until an instruction to terminate is received from the user.

[0178] As shown in FIG. 12, the image processing device 100 acquires the input image 10 from the imaging device 200 (step S201).

[0179] Next, the image processing device 100 generates the nth (n is an integer equal to or greater than 2) image group (nth block 20_n) (step S202). The nth block 20_n includes the input image 10 acquired in step S201 and a portion of the input image 10 included in the (n-1)th block 20_n-1.

[0180] Next, the image processing apparatus 100 adds the input images 10_n included in the n-th block 20_n to generate the n-th added image (n-th added image 30_n) (step S203).

[0181] The image processing apparatus 100 acquires the (n-1)th added image (the (n-1)th added image 30_n-1) (step S204).

[0182] The image processing device 100 performs motion estimation between the n-th added image 30_n and the (n-1)-th added image 30_n-1 (step S205). The image processing device 100 calculates a correction value for each input image 10 included in the n-th block 20_n according to the motion estimation result (step S206).

[0183] The image processing device 100 corrects each of the input images 10 included in the n-th block 20_n, and aligns the positions of the input images 10 (step S207).

[0184] The image processing device 100 adds the corrected input image 10 (corrected input image) to generate a corrected added image 31 (n-th corrected added image 31_n) (step S208).

[0185] The image processing apparatus 100 acquires the (n-1)th composite image (the (n-1)th composite image 40_n-1) (step S209).

[0186] The image processing device 100 performs motion estimation between the n-th corrected added image 31_n and the (n-1)th composite image 40_n-1 (step S210). The image processing device 100 corrects the (n-1)th composite image (the (n-1)th composite image 40_n-1) according to the result of the motion estimation, and generates the (n-1)th corrected composite image 41_n-1 (step S211).

[0187] The image processing apparatus 100 combines the n-th corrected added image 31_n and the (n-1)th combined image 40_n-1 to generate the n-th combined image 40 (n-th combined image 40_n) (step S212).

[0188] The image processing apparatus 100 determines whether the signal-to-noise ratio of the generated n-th composite image 40 (n-th composite image 40_n) exceeds a threshold value (predetermined threshold value) (step S213).

[0189] If the S / N ratio of the n-th composite image 40_n does not exceed the threshold, that is, if the S / N ratio is equal to or less than the threshold (step S213; No), the image processing apparatus 100 ends the process.

[0190] On the other hand, if the S / N ratio of the nth composite image 40_n exceeds the threshold value (step S213; Yes), the image processing device 100 outputs the nth composite image 40 (nth composite image 40_n), for example, to the display unit 120 (step S214), and terminates the processing.

[0191] As described above, the image processing device 100 according to the embodiment of the present disclosure includes the addition unit 132, the correction unit 133, and the synthesis unit 134 (an example of a generation unit).

[0192] The addition unit 132 generates a current added image (e.g., the nth added image 30_n) by adding together multiple input images 10 included in the current image group (e.g., the nth block 20_n), which are captured consecutively.

[0193] The correction unit 133 generates the current corrected added image (e.g., the nth corrected added image 31_n) by correcting and adding multiple input images 10 according to the deviation between the previous added image (e.g., the n-1th added image 30_n-1) and the current added image.

[0194] The synthesis unit 134 generates the current synthesis image (e.g., the nth synthesis image 40_n) based on the previous synthesis image (e.g., the n-1th synthesis image 40_n-1) and the current corrected addition image (e.g., the nth corrected addition image 31_n).

[0195] In this way, the image processing device 100 generates the current composite image 40 using the results of generating the previous composite image 40 (for example, the previous composite image 40 and the previous added image 30). This allows the image processing device 100 to recommend the process of generating the added image 30 and the process of correcting the input image 10, thereby reducing the amount of processing (calculation amount) in the reconstruction process and improving the processing speed.

[0196] Therefore, the image processing device 100 can apply the reconstruction process to moving images as well. As described above, the image processing device 100 according to this embodiment can execute the reconstruction process at a higher speed. Therefore, the image processing device 100 can perform the reconstruction process on moving images in real time.

[0197] Furthermore, the image processing device 100 can reduce the number of images used in the reconstruction process by generating the current composite image 40 using the results of the previous generation of the composite image 40 (for example, the previous composite image 40 and the previous added image 30). This allows the image processing device 100 to reduce the frame memory required for the reconstruction process.

[0198] Furthermore, the multiple input images 10 included in the current image group (e.g., the nth block 20_n) partially overlap with the multiple input images 10 included in the previous image group (e.g., the n-1th block 20_n-1).

[0199] Therefore, the range of motion between the current added image 30 obtained by adding the current image group (for example, the nth added image 30_n) and the previous added image 30 obtained by adding the previous image group (for example, the (n-1)th added image 30_n-1) is reduced. Similarly, the range of motion between the corrected added image 31 and the previous composite image 40 is reduced. This allows the image processing device 100 to reduce the amount of calculation required for motion estimation between these images.

[0200] <<3. Other Embodiments>> The processing according to the above-described embodiment may be implemented in various different forms other than the above-described embodiment.

[0201] For example, in the above embodiment, motion estimation is performed between the n-th corrected and added image 31_n and the (n-1)-th composite image 40_n-1, but this motion estimation may be omitted.

[0202] Fig. 13 is a diagram showing an example of a reconstruction process according to another embodiment of the present disclosure. The reconstruction process in Fig. 13 is the same as the reconstruction process shown in Fig. 8 and Fig. 9 except that the image processing device 100 does not perform motion estimation between the n-th corrected added image 31_n and the (n-1)-th composite image 40_n-1.

[0203] The image processing device 100 corrects the (n-1)th composite image 40_n-1 based on the result of motion estimation between the nth and (n-1)th added images 30_n and 30_n-1, and generates the (n-1)th corrected composite image 41_n-1.

[0204] The image processing device 100 generates the nth composite image 40_n by combining the (n-1)th corrected composite image 41_n-1 and the nth corrected added image 31_n.

[0205] In this way, the (n-1)th composite image 40_n-1 is corrected based on the result of motion estimation between the nth and (n-1)th added images 30_n and 30_n-1. This allows the image processing device 100 to omit motion estimation processing between the nth corrected added image 31_n and the (n-1)th composite image 40_n-1, thereby reducing the number of times motion estimation processing is performed.

[0206] Fig. 14 is a diagram showing another example of reconstruction processing according to another embodiment of the present disclosure. The reconstruction processing in Fig. 14 is the same as the reconstruction processing in Fig. 13 in that the (n-1)th composite image 40_n-1 is corrected based on the result of motion estimation between the nth and (n-1)th added images 30_n and 30_n-1.

[0207] The reconstruction process of Figure 14 differs from the reconstruction process of Figure 13 in that when the (n-1)th corrected composite image 41_n-1 is synthesized with the nth corrected added image 31_n, the results of motion estimation between the nth and (n-1)th added images 30_n and 30_n-1 are taken into consideration.

[0208] For example, the image processing device 100 weights the (n-1)th corrected composite image 41_n-1 according to the result of motion estimation between the nth and (n-1)th added images 30_n and 30_n-1, and combines it with the nth corrected added image 31_n.

[0209] For example, if the image processing device 100 estimates that there is a large amount of movement between the nth and n-1th added images 30_n and 30_n-1, it weights the n-1th corrected composite image 41_n-1 so that the addition (combination) ratio of the n-1th corrected composite image 41_n-1 becomes smaller.

[0210] That is, the image processing device 100 weights the n-1th corrected composite image 41_n-1 less as the motion estimation value (amount of motion) resulting from motion estimation between the nth and n-1th added images 30_n and 30_n-1 becomes larger.

[0211] In this way, the image processing device 100 weights the (n-1)th corrected composite image 41_n-1 according to the motion estimation value between the nth and (n-1)th added images 30_n and 30_n-1, and combines it with the nth corrected added image 31_n to generate the nth composite image 40_n.

[0212] If the motion estimation value is large, that is, if the subject moves significantly between the nth and (n-1)th added images 30_n and 30_n-1, the accuracy of the motion estimation will deteriorate. In other words, if the motion estimation value is large, the probability of errors occurring in the motion estimation will increase.

[0213] In this way, when the motion estimation value is large, the image processing device 100 reduces the addition (composition) ratio of the n-1th corrected composite image 41_n-1 (i.e., the rate at which the n-1th corrected composite image 41_n-1 is added) and combines the n-1th corrected composite image 41_n-1 and the nth corrected added image 31_n.

[0214] As a result, even if an error occurs in the motion estimation, the image processing device 100 can reduce the influence of the (n-1)th corrected composite image 41_n-1 on the nth composite image 40_n.

[0215] Here, it is assumed that the image processing device 100 combines the n-1th corrected composite image 41_n-1 with the nth corrected added image 31_n at an addition ratio according to the motion estimation result between the nth and n-1th added images 30_n and 30_n-1.

[0216] Alternatively, for example, the image processing device 100 may switch between the reconstruction process of Figures 8 and 9 and the reconstruction process of Figure 13 depending on the motion estimation result between the nth and n-1th added images 30_n and 30_n-1.

[0217] For example, if the motion estimation value between the nth and n-1th added images 30_n and 30_n-1 is equal to or less than a predetermined threshold, the image processing device 100 executes the reconstruction process of Fig. 13. That is, the image processing device 100 corrects the (n-1)th composite image 40_n-1 based on the result of the motion estimation between the nth and (n-1)th added images 30_n and 30_n-1.

[0218] For example, if the motion estimation value between the nth and (n-1)th added images 30_n and 30_n-1 exceeds a predetermined threshold, the image processing device 100 executes the reconstruction processing of Fig. 8 and Fig. 9. That is, the image processing device 100 corrects the (n-1)th composite image 40_n-1 in accordance with the motion estimation result between the nth corrected added image 31_n and the (n-1)th composite image 40_n-1.

[0219] In this way, the image processing device 100 may switch the reconstruction process to be performed depending on the motion estimation value between the nth and (n-1)th added images 30_n and 30_n-1.

[0220] In the above-described embodiment, the imaging device 200 and the image processing device 100 are mounted in an electronic device 300 such as a camera or a smartphone (see FIG. 7 ). The device in which the image processing device 100 is mounted is not limited to the electronic device 300.

[0221] For example, the image processing device 100 may be mounted on a device (e.g., an information processing device) different from the electronic device 300. In this case, the image processing device 100 may be connected wirelessly or with a wire to, for example, the imaging device 200 (or the electronic device 300 in which the imaging device 200 is mounted).

[0222] Alternatively, the image processing device 100 may be installed in a server device. In this case, the image processing device 100 may be connected to, for example, the imaging device 200 (or the electronic device 300 in which the imaging device 200 is installed) via a network such as the Internet.

[0223] Furthermore, the image processing device 100 may be mounted on the imaging device 200. For example, the image processing device 100 may be mounted on a logic chip of the imaging device 200 having a stacked structure.

[0224] 15 is a diagram illustrating an example of a stack structure of an imaging device 200 according to another embodiment of the present disclosure. The imaging device 200 has a stack structure in which a semiconductor substrate 201 called a SPAD chip and a semiconductor substrate 202 called a logic chip are stacked one above the other.

[0225] The SPAD chip (semiconductor substrate 201) is, for example, a semiconductor chip having a pixel array section (not shown) in which photodiodes (image pickup elements) PD are arranged. The logic chip (semiconductor substrate 202) is, for example, a semiconductor chip having pixel circuits that read pixel signals and peripheral circuits that process the pixel signals. The image processing device 100 according to this embodiment can be mounted on this logic chip (semiconductor substrate 202).

[0226] The two semiconductor substrates 201 and 202 can be bonded together by, for example, flattening the bonding surfaces of the substrates and bonding them together using atomic force, which is called direct bonding. However, the embodiments of the present disclosure are not limited to this, and other bonding methods such as Cu-Cu bonding, which involves bonding together copper (Cu) electrode pads formed on the bonding surfaces of the substrates, or bump bonding can also be used.

[0227] The two semiconductor substrates 201 and 202 are electrically connected via a connection portion such as a TSV (Through-Silicon Via) that penetrates the semiconductor substrates.

[0228] For connection using TSVs, for example, the so-called twin TSV method can be adopted, in which two TSVs, one provided in semiconductor substrate 201 and the other provided from semiconductor substrate 201 to semiconductor substrate 202, are connected on the outer surface of the chip.

[0229] Alternatively, for connection using a TSV, for example, a so-called shared TSV method can be adopted in which the two are connected by a TSV that penetrates from the semiconductor substrate 201 to the semiconductor substrate 202 .

[0230] However, when Cu--Cu bonding or bump bonding is used to bond the two semiconductor substrates 201 and 202, they are electrically connected via the Cu--Cu bonding portion or the bump bonding portion.

[0231] In the embodiment of the present disclosure, the stacked structure of chips in the imaging device 200 is not limited to the example shown in Fig. 15. For example, the imaging device 200 may have a stacked structure of three or more layers.

[0232] Furthermore, the image processing device 100 may change the number of input images 10 included in the block 20 while repeatedly executing the reconstruction process. For example, the image processing device 100 may change the number of input images 10 included in the block 20 according to the pixel values ​​of the composite image 40 to be generated, the S / N ratio of the composite image 40, or the like.

[0233] Specifically, the image processing device 100 increases the number of input images 10 included in the block 20 as the average value (or S / N ratio) of the pixel values ​​of the composite image 40 becomes smaller, and decreases the number of input images 10 as the average value (or S / N ratio) becomes larger.

[0234] Alternatively, the image processing device 100 may change the number of input images 10 included in the block 20 according to the pixel values ​​of the input images 10 or the signal-to-noise ratio of the input images 10 .

[0235] The number of input images 10 included in the 0th and first blocks 20_0 and 20_1 may be a predetermined value, or may be determined according to the brightness of the surroundings of the imaging device 200, or the like.

[0236] In addition, the number of overlapping input images 10 of adjacent blocks 20 (e.g., the nth and n-1th blocks 20_n and 20_n-1), in other words, the amount of shift between adjacent blocks 20, may be changed depending on the result of motion estimation (motion estimation value).

[0237] For example, the image processing device 100 increases the amount of deviation between the blocks 20 as the motion estimation value (or correction amount) becomes smaller, and decreases the amount of deviation between the blocks 20 as the motion estimation value (or correction amount) becomes larger.

[0238] The motion estimation value that serves as an index for changing the amount of deviation may be the result of motion estimation between the added images 30, or may be the result of motion estimation between the current corrected added image 31 and the previous composite image 40. Alternatively, it may be the average value of these motion estimation results.

[0239] <<4. Supplementary Information>> Although preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0240] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0241] Note that the present technology can also be configured as follows. (1) An image processing device comprising: an adder that generates a current added image by adding together a plurality of input images captured consecutively, the input images being included in a current image group; a correction unit that generates a current corrected added image by correcting and adding together the plurality of input images according to a deviation between the previous added image and the current added image; and a generator that generates the current composite image based on the previous composite image and the current corrected added image. (2) The image processing device described in (1), in which the generator combines a previous corrected composite image obtained by performing motion correction on the previous composite image with the current corrected added image to generate the current composite image. (3) The image processing device described in (1) or (2), in which the input images included in the current image group and the input images included in the previous image group partially overlap. (4) The image processing device according to any one of (1) to (3), wherein the correction unit calculates a correction value for each of the plurality of input images from the deviation between the previous added image and the current added image, and corrects each of the plurality of input images according to the correction value. (5) The image processing device according to any one of (1) to (4), wherein the correction unit corrects each of the plurality of input images by using one of the plurality of input images as a reference and aligning the remaining input images. (6) The image processing device according to any one of (1) to (5), wherein the generation unit calculates a previous corrected composite image by aligning the previous composite image using the current corrected added image as a reference. (7) The image processing device according to any one of (1) to (6), wherein the generation unit calculates a previous corrected composite image according to the deviation between the current corrected added image and the previous composite image. (8) The image processing device according to any one of (1) to (6), wherein the generation unit calculates a previous corrected composite image in accordance with the deviation between the previous added image and the current added image. (9) The image processing device according to (8), wherein the generation unit combines the previous corrected composite image with the current corrected added image at a ratio in accordance with the deviation between the previous added image and the current added image.(10) The image processing device according to any one of (1) to (9), further comprising a storage unit that stores at least one of the previous added image and the previous composite image. (11) An image processing method including: generating a current added image by adding together a plurality of input images included in a current image group that have been captured consecutively, generating a current corrected added image by correcting and adding together the plurality of input images according to a deviation between the previous added image and the current added image, and generating the current composite image based on the previous composite image and the current corrected added image.

[0242] REFERENCE SIGNS LIST 100 Image processing device 110 Storage unit 120 Display unit 130 Control unit 131 Acquisition unit 132 Addition unit 133 Correction unit 134 Combination unit 200 Imaging device 300 Electronic device

Claims

1. An image processing device comprising: an adder that generates a current added image by adding together a plurality of input images captured consecutively, the input images being included in a current image group; a correction unit that generates a current corrected added image by correcting and adding together the plurality of input images according to a deviation between the previous added image and the current added image; and a generation unit that generates the current composite image based on the previous composite image and the current corrected added image.

2. The image processing device according to claim 1, wherein the generation unit generates the current composite image by combining a previous corrected composite image obtained by performing motion correction on the previous composite image with the current corrected added image.

3. The image processing device according to claim 1, wherein the plurality of input images included in the current image group and the plurality of input images included in the previous image group partially overlap.

4. The image processing device according to claim 1, wherein the correction unit calculates a correction value for each of the plurality of input images from the deviation between the previous added image and the current added image, and corrects each of the plurality of input images according to the correction value.

5. The image processing device according to claim 1, wherein the correction unit corrects each of the plurality of input images by using one of the plurality of input images as a reference and aligning the remaining input images.

6. The image processing device according to claim 1, wherein the generation unit calculates the previous corrected composite image by aligning the previous composite image with the current corrected added image as a reference.

7. The image processing device according to claim 1, wherein the generation unit calculates a previous corrected composite image in accordance with a deviation between the current corrected added image and the previous composite image.

8. The image processing device according to claim 1, wherein the generation unit calculates a previous corrected composite image in accordance with the deviation between the previous added image and the current added image.

9. The image processing device according to claim 8, wherein the generation unit combines the previous corrected composite image with the current corrected added image at a ratio according to the deviation between the previous added image and the current added image.

10. The image processing device according to claim 1, further comprising a storage unit for storing at least one of the previous added image and the previous composite image.

11. An image processing method including: generating a current added image by adding together a plurality of input images included in a current image group that have been captured consecutively; generating a current corrected added image by correcting and adding together the plurality of input images according to a deviation between the previous added image and the current added image; and generating the current composite image based on the previous composite image and the current corrected added image.

Images