Noise reduction device, noise reduction method, and computer program

Abstract

One embodiment of the present invention is a noise reduction device provided with: a provisional image reconstruction unit that generates a provisional image by reconstructing an encoded image obtained by compressed spectral imaging; a similar patch search unit that obtains a plurality of similar provisional patches, which are small regions containing images similar to each other, in the provisional image; a group generation unit that obtains a plurality of similar encoded patches, which are small regions containing images similar to each other, in the encoded image on the basis of information about the positions at which the similar provisional patches are obtained; a low-rank approximation unit that, on the basis of the plurality of similar encoded patches, performs low-rank approximation to reduce the noise in the images of regions contained in the similar encoded patches, thus obtaining patches with reduced noise; and a patch integration unit that integrates the plurality of patches with reduced noise according to information about the positions of the patches to generate an encoded image.

Description

[Technical Field]

[0001] The present invention relates to a noise reduction device, a noise reduction method, and a computer program technology. [Background technology]

[0002] Typical spectral imaging devices use a complex imaging device consisting of a slit and a dispersive element to mechanically move the slit and capture multiple images to obtain a single spectral image. This makes it difficult to capture video, which requires capturing dozens of frames per second. Compressed spectral imaging, on the other hand, utilizes compressed sensing theory to enable the acquisition of spectral images at video-level frame rates. Compressed sensing is a sensing theory that exploits the statistical properties (redundancy) of the signal being sensed, enabling the acquisition of target signals with fewer samples than the sampling theorem would dictate. In compressed spectral imaging, a spectral image is optically encoded and then captured by a camera in color or monochrome format. Image reconstruction technology is then used to estimate spectral information from the encoded image, resulting in a spectral image.

[0003] The encoded images in compressed spectral imaging may contain noise, such as thermal noise and shot noise, just like general image capture. When the exposure time is shortened to achieve a high frame rate, the amount of noise increases relatively. Reconstructing a spectral image from an encoded image containing a large amount of noise can result in failure to restore fine features or artifacts (significant distortions caused by the reconstruction process). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. “Weighted nuclear norm minimization with application to image denoising.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2862-2869, 2014. Summary of the Invention [Problem to be solved by the invention]

[0005] To solve this problem, it is possible to apply noise reduction processing to coded images. However, if noise reduction methods for color or monochrome images or videos are directly applied, they will not work effectively because the properties of coded images are significantly different from those of general images. The extent to which the properties differ from general monochrome or color images depends on the method of compressed spectral imaging.

[0006] For example, a method called CASSI uses a coded aperture in the imaging device. With this method, the aperture pattern on the mosaic may be observed overlapping with the object being imaged in the coded image. Furthermore, with wavelength-dependent PSF methods, the image is significantly different from general images in that it can be significantly blurred or overlapped. Thus, coded images have significantly different properties from general images. Furthermore, the properties of coded images also differ significantly depending on the compressed spectral imaging method. This has made it difficult to properly reduce noise.

[0007] In view of the above circumstances, an object of the present invention is to provide a technique capable of reducing noise in coded images obtained by compressed spectral imaging. [Means for solving the problem]

[0008] One aspect of the present invention is a noise reduction device comprising: a provisional image reconstruction unit that generates a provisional image by reconstructing an encoded image obtained by compressed spectral imaging; a similar patch search unit that acquires a plurality of similar provisional patches in the provisional image, which are small regions containing images that are similar to each other; a group generation unit that acquires a plurality of similar encoded patches, which are small regions in the encoded image containing images that are similar to each other, based on information about the positions at which the similar provisional patches were acquired; a low-rank approximation unit that performs low-rank approximation based on the plurality of similar encoded patches to acquire patches in which noise on the image has been reduced in areas included in the similar encoded patches; and a patch integration unit that generates an encoded image by integrating a plurality of noise-reduced patches according to information about the positions of the patches.

[0009] One aspect of the present invention is a noise reduction method comprising: a tentative image reconstruction step of generating a tentative image by reconstructing an encoded image obtained by compressed spectral imaging; a similar patch search step of acquiring a plurality of similar temporary patches in the tentative image, which are small regions containing images similar to each other; a group generation step of acquiring a plurality of similar encoded patches, which are small regions in the encoded image containing images similar to each other, based on information about the positions at which the similar temporary patches were acquired; a low-rank approximation step of performing low-rank approximation based on a plurality of the similar encoded patches to acquire patches in which noise on the image has been reduced in regions included in the similar encoded patches; and a patch integrating step of generating an encoded image by integrating a plurality of noise-reduced patches according to information about the positions of the patches.

[0010] One aspect of the present invention is a computer program for causing a computer to function as the noise reduction device described above. [Effects of the Invention]

[0011] The present invention makes it possible to reduce noise in coded images obtained by compressed spectral imaging. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an overview of the technology of the present invention. [Figure 2] 1 is a diagram illustrating an example of the configuration of a spectral image generating device 100 according to the present invention. [Figure 3] FIG. 2 is a diagram illustrating a first embodiment of the noise reduction unit 21. [Figure 4] FIG. 2 is a diagram showing a specific example of the processing flow of the spectral image generating device 100 including the noise reduction unit 21 of the first embodiment. [Figure 5] FIG. 10 is a diagram showing a second embodiment of the noise reduction section 21 (noise reduction section 21a). [Figure 6] FIG. 10 is a diagram showing a specific example of the flow of processing in the spectral image generating device 100 including the noise reduction unit 21a of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] [Summary] Embodiments of the present invention will be described in detail with reference to the drawings. First, an outline of an embodiment of the present invention will be described. In the following description, characters represented using "_" in the text as follows indicate subscripts. For example, the notation "M_i" indicates that an "i" is added as a subscript to the lower right of "M". For example, the notation "M_(i+1)" indicates that an "i+1" is added as a subscript to the lower right of "M". In the following description, characters represented using "^" in the text as follows indicate characters with a "^" added above them. For example, the notation "M^" indicates that a "^" is added above "M". In the following description, characters represented using "-" in the text as follows indicate superscripts. For example, the notation "Mi" indicates that an "i" is added as a superscript to the upper right and above "M". For example, the notation "M-(i+1)" indicates that an "i+1" is added as a superscript to the upper right and above "M".

[0014] FIG. 1 is a diagram illustrating an overview of the technology of the present invention. This embodiment provides a noise reduction method for coded images of compressed spectral imaging using a low-rank matrix restoration method based on weighted nuclear norm minimization. In the low-rank matrix restoration method based on nuclear norm minimization (see Non-Patent Document 1), an image is spatially divided into multiple small regions (hereinafter referred to as "patches") of a predetermined size. Then, patches containing similar image patterns (hereinafter referred to as "similar patches") must be acquired from within the same frame or from different frames. However, coded images of compressed spectral imaging are special images that are optically encoded. Therefore, commonly used techniques such as block matching cannot accurately acquire similar patches.

[0015] Therefore, in this embodiment, a spectral image is first tentatively reconstructed for searching for similar patches in an encoded image 80 before noise reduction. Then, a plurality of patches 91 (hereinafter referred to as "temporary patches") are generated using the tentatively reconstructed spectral image (hereinafter referred to as "temporary image") 90, and positions of similar temporary patches 92 that are similar to each other are determined from the tentative patches 91. The similar temporary patches 92 may be searched for within the same provisional image or may be searched for in other provisional images at different times. In this embodiment, the similar temporary patches 92 are searched for in both provisional images. In the example of FIG. 1 , similar temporary patches 92 are searched for in provisional images at three different times. The position of each similar temporary patch 92 is represented using spatial information (e.g., information on the position expressed by spatial coordinates within the image) and temporal information (e.g., information on the time of capture and information on the sequential frame numbers).

[0016] The positions of similar temporary patches 92 obtained using the temporary image 90 are used to identify the positions of patches (hereinafter referred to as "similar coded patches") 81 in the coded image 80 that are similar to each other. Noise reduction is performed on the coded image using the similar coded patches 81. The quality of the temporary image 90 is low because it uses the coded image 80 that contains noise. However, it can be used to search for the positions of the similar temporary patches 92. In this way, by using the temporary image 90 to collect the similar coded patches 81 in the coded image 80, it is possible to prevent inaccuracies in collecting similar patches due to the influence of properties specific to the coded image.

[0017] [detail] Next, the technology of the present invention will be described in detail. Fig. 2 is a diagram showing an example of the configuration of a spectral image generation device 100 of the present invention. The spectral image generation device 100 includes an encoded image acquisition unit 10, a control unit 20, and an encoded image storage unit 30.

[0018] The encoded image acquisition unit 10 acquires an encoded spectral image (hereinafter referred to as an "encoded image"). The encoded image acquisition unit 10 may acquire data of an encoded image that has been captured in advance, or may acquire data of an encoded image by capturing an image. For example, if the spectral image to be acquired is x, then x can be expressed by the following equation 1.

[0019]

number

[0020] In Equation 1, H, W, and Λ represent the number of elements on the vertical axis, horizontal axis, and spectral axis, respectively. The encoded image acquisition unit 10 may capture such a spectral image of the acquisition target as a color or monochrome encoded image y through an optical system. In this case, y can be expressed by the following Equation 2.

[0021]

number

[0022] In Equation 1 and Equation 2, the relationship between the symbols is expressed as in Equation 3 below.

[0023]

number

[0024] The ill-posed problem that arises when estimating a spectral image x from an encoded image y is called a reconstruction problem. The optical observation process can be expressed as y = Φx using the following equation 4.

[0025]

number

[0026] The optical observation process for converting a spectral image x into an encoded image y may be any method of compressed spectral imaging. The encoded image may be an encoded image captured at a single timing on the time axis acquired by the encoded image acquisition unit 10, or may be a time-series encoded image that is discretely continuous on the time axis. In the following description, an example in which a time-series encoded image is acquired will be described.

[0027] The control unit 20 is configured using a processor such as a CPU (Central Processing Unit) and a memory. The control unit 20 functions as a noise reduction unit 21 and an image reconstruction unit 22 when the processor executes a program. All or part of the functions of the control unit 20 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., a solid-state drive (SSD)), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The program may be transmitted via a telecommunications line.

[0028] The noise reduction unit 21 performs noise reduction processing on the coded images acquired by the coded image acquisition unit 10. The noise reduction unit 21 uses one or more coded images y to perform noise reduction processing on at least one of the coded images. In the following description, the noise reduction unit 21 performs noise reduction processing on at least one coded image using three coded images y_(i-1), y_i, and y_(i+1) in time series. Note that the number of coded images used in the noise reduction processing may be more than three, or may be one or two. Details of the configuration of the noise reduction unit 21 and the details of the noise reduction processing will be described later. Note that the noise reduction unit 21 corresponds to a noise reduction device. In this case, the noise reduction device is configured as an information device (information processing device) equipped with the above-mentioned processor, etc.

[0029] The image reconstructing unit 22 reconstructs a spectral image based on the data of the encoded image in which noise has been reduced by the noise reducing unit 21. The image reconstructing unit 22 outputs the data of the reconstructed spectral image.

[0030] The encoded image storage unit 30 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The encoded image storage unit 30 stores data of the encoded image in which noise has been reduced by the noise reduction unit 21. By using the data stored in the encoded image storage unit 30, it is possible to reconstruct a spectral image in which noise has been reduced.

[0031] [First embodiment of noise reduction unit] 3 is a diagram showing a first embodiment of the noise reduction unit 21. The noise reduction unit 21 includes one or more provisional image reconstruction units 211, a similar patch search unit 212, a group generation unit 213, a low-rank approximation unit 214, and a patch integration unit 215. In this embodiment, as described above, three coded images in time series are input. It is desirable that the three coded images to be input are close in time. For example, a certain coded image (a target image in the following description), an coded image generated immediately before that, and an coded image generated immediately after that may be input.

[0032] In this embodiment, noise is reduced in one of three input coded images. For example, noise may be reduced in the middle coded image in chronological order among three input coded images in chronological order. In the following description, the coded image for which noise reduction is performed is referred to as the "target image." Of the multiple input coded images, coded images other than the target image are referred to as "auxiliary images." If there are multiple auxiliary images, they are referred to in chronological order (earliest to earliest) as the "first auxiliary image," the "second auxiliary image," etc., in order to make them identifiable. Note that the noise reduction process is repeatedly performed on multiple coded images. A coded image treated as an auxiliary image in one process is also treated as a target image in another process, thereby reducing noise. However, noise reduction does not necessarily have to be performed on all coded images. For example, coded images in a predetermined cycle may be treated as target images, thereby obtaining coded images in which noise is reduced at a predetermined cycle.

[0033] The three time-sequentially input encoded images are each input to the provisional image reconstruction unit 211. The provisional image reconstruction unit 211 generates a spectral image by performing a reconstruction process on the input encoded images (target image, auxiliary image). The spectral image reconstruction method executed in the provisional image reconstruction unit 211 may be any compressed sensing reconstruction method. In this case, for example, the value of Φ in the above-mentioned equation 4 is required. The provisional image reconstruction unit 211 may have the value of Φ in advance, or Φ may be input to the provisional image reconstruction unit 211 together with the encoded images. The spectral images generated by the provisional image reconstruction unit 211 have not had noise reduced. The spectral images generated by the provisional image reconstruction unit 211 are called "provisional images."

[0034] The similar patch search unit 212 divides each provisional image generated by the provisional image reconstruction unit 211 into multiple provisional patches. The areas of the multiple provisional patches may overlap within the same provisional image. For each patch of the provisional image generated from the target image (hereinafter referred to as a "provisional target patch"), the similar patch search unit 212 acquires other provisional patches with high similarity. For example, the similar patch search unit 212 evaluates the similarity of each provisional target patch with other provisional patches that satisfy predetermined conditions indicating spatial and temporal proximity by block matching, and collects the top K provisional patches. The value of K is a predetermined integer equal to or greater than 1. The similar patch search unit 212 outputs position information (spatial information and temporal information) of each of the collected top K provisional patches as similar patch information. The matching measure may be any measure, such as mean square error.

[0035] The group generation unit 213 receives the input encoded image (y_(i-1), y_i, y_(i+1)) and similar patch information, and outputs similar patch group information Y_j. The group generation unit 213 finds spatially and temporally corresponding patches in the encoded image based on information about the positions of the similar patch information in the provisional image. The group generation unit 213 then extracts similar encoded patches from the encoded image, and outputs a similar patch group including these patches. For example, the extracted patches may be treated as column vectors, which are then stacked and formed into a matrix to generate the similar patch group Y_j.

[0036] The low-rank approximation unit 214 receives the similar patch group Y_j as input and generates a low-ranked Y^_j. The low-rank approximation unit 214 may use, for example, the WNNM described in Patent Document 1 mentioned above, or may use MC-WNNM, which can effectively utilize redundancy between colors.

[0037] The low-rank approximation unit 214 first obtains a weight vector w. The weight vector w can be expressed, for example, as shown in Equation 5 below.

[0038]

number

[0039] Here, σ_i(Y_j) is the i-th singular value of Y_j, c is a positive real constant, σ_n is the variance of noise, and ε is a small positive real number to avoid division by zero. Next, the low-rank approximation unit 214 performs singular value decomposition on Y_j.

[0040]

number

[0041] Then, the low-rank approximation unit 214 performs soft thresholding using the weight vector w on the singular values to reduce the rank.

[0042]

number

[0043] S_w(Σ) is the soft thresholding process using w as the threshold.

[0044]

number

[0045] The patch merging unit 215 generates a noise-reduced encoded image using the similar patch group Y^_j in which noise has been reduced by low-rank approximation. The patch merging unit 215 obtains the spatial and temporal position of each patch (each column vector) of Y^_j before patch division from the similar patch information, and then merges the divided patches to reconstruct an encoded image of the original size. Through this process, the patch merging unit 215 outputs data of a target image in which noise has been reduced (hereinafter referred to as a "noise-free target image").

[0046] 4 is a diagram showing a specific example of the processing flow of the spectral image generation device 100 including the noise reduction unit 21 of the first embodiment. First, the encoded image acquisition unit 10 acquires encoded images to be used in noise reduction processing (step S11). For example, if three encoded images are used in one processing by the noise reduction unit 21 as described above, the encoded image acquisition unit 10 may acquire three encoded images. The tentative image reconstruction unit 211 of the noise reduction unit 21 reconstructs a tentative image for each input encoded image (step S12).

[0047] The similar patch searching unit 212 obtains a plurality of temporally and spatially spread temporary patches by dividing each temporary image into patches (step S13). The similar patch searching unit 212 searches for similar temporary patches that are similar to each other (step S14). For example, the similar patch searching unit 212 may search for a predetermined number of temporary patches that have a high degree of similarity to the temporary target patch that is the processing target as similar temporary patches. Through such processing, similar patch information is generated.

[0048] The group generation unit 213 finds similar coded patches that correspond spatially and temporally in the coded image based on the similar patch information generated in the provisional image, and generates a similar patch group (step S15). The low-rank approximation unit 214 performs low-rank approximation on the generated similar patch group to generate a plurality of similar coded patches with reduced noise (step S16). The patch integration unit 215 integrates a plurality of similar coded patches with reduced noise based on spatial and temporal information to generate a noise-reduced coded image. For example, the patch integration unit 215 generates a noise-reduced coded image (non-noise target image) for the target image (step S17). The image reconstruction unit 22 generates a noise-reduced spectral image by reconstructing the non-noise target image (step S18).

[0049] [Second embodiment of noise reduction unit] FIG. 5 is a diagram illustrating a second embodiment (noise reduction unit 21a) of the noise reduction unit 21. The noise reduction unit 21a of the second embodiment differs from the noise reduction unit 21 of the first embodiment in that it further includes an iterative processing control unit 216. Furthermore, while the group generation unit 213 of the first embodiment generates similar coded patches from an input coded image, the similar patch search unit 212 of the second embodiment generates similar coded patches using coded images generated by the patch merging unit 215 in the iterative processing (processing from the second cycle onward). Apart from these points, the noise reduction unit 21 of the first embodiment and the noise reduction unit 21a of the second embodiment have the same configuration. Therefore, the noise reduction unit 21a of the second embodiment may perform low-rank approximation not only on the input target image but also on an auxiliary image, and may perform patch merging to generate a coded image in which noise has been reduced. This configuration enables more efficient noise reduction in the iterative processing.

[0050] The iterative process control unit 216 controls the iterative processes in the group generation unit 213, the low-rank approximation unit 214, and the patch integration unit 215. For example, the iterative process control unit 216 determines and controls whether to further iterate the processes in these functional units or to terminate them without iterating. The iterative process control unit 216 may continue the iterative process until a predetermined iteration termination condition is met (while the condition is not met), and may terminate the iterative process when the iteration termination condition is met. As a specific example of the iteration termination condition, for example, a value indicating a predetermined number of iterations may be set, or a threshold value may be set for the amount of change in one step of the iterative process.

[0051] 6 is a diagram showing a specific example of the processing flow of the spectral image generating device 100 including the noise reduction unit 21a of the second embodiment. In the processing of FIG. 6, compared to the processing of FIG. 4, a branching process of step S21 is provided after the processing of step S16. After the processing of steps S11 to S16 is performed, the iterative processing control unit 216 determines whether or not the iterative end condition is satisfied (step S21). If the iterative end condition is not satisfied (step S21-NO), the processing of steps S15 to S21 is repeatedly executed. On the other hand, if the iterative end condition is satisfied (step S21-YES), the processing of steps S17 and S18 is executed.

[0052] The spectral image generation device 100 (including the first and second embodiments) configured as described above searches for the spatial and temporal positions of similar patches in a spectral image (provisional image) tentatively reconstructed from an encoded image. Based on the search results, corresponding patches are acquired in the encoded image and subjected to low-rank approximation. Then, a plurality of low-rank approximated patches are integrated according to their spatial and temporal positions to construct an encoded image, generating a noise-reduced encoded image. Since the spatial and temporal positions of similar patches (similar provisional patches) in the provisional image are determined in this manner, similar patches (similar encoded patches) can be searched for with greater accuracy. As a result, noise can be reduced more accurately in encoded images obtained by compressive spectral imaging. Specifically, searching for similar patches in an encoded image may result in inaccurate grouping due to the influence of Φ. However, in this embodiment, similar patches are searched for in the provisional image, enabling more accurate grouping by eliminating the influence of Φ.

[0053] Furthermore, the above-described processing makes it possible to reduce noise in the encoded image without relying on the compressed spectral imaging method.

[0054] Furthermore, in the spectral image generating device 100 including the noise reduction unit 21a of the second embodiment, it is possible to generate an encoded image with further reduced noise by performing iterative processing.

[0055] (Variation) In the noise reduction unit 21 of the first embodiment, noise reduction may be performed on some or all of the input encoded images, rather than only on the target image among the input encoded images.

[0056] In the noise reduction unit 21 of the second embodiment, noise reduction may be performed on only some of the input encoded images, rather than on all of the input encoded images.

[0057] In the noise reduction unit 21 of the first and second embodiments, only one encoded image may be input, or two or four or more encoded images may be input. In the noise reduction unit 21 of the first and second embodiments, the number of tentative image reconstruction units 211 does not necessarily have to match the number of input encoded images (three in the above specific example). If the number of tentative image reconstruction units 211 is smaller than the number of input encoded images, one tentative image reconstruction unit 211 may perform reconstruction processing on multiple encoded images in one noise reduction to generate multiple tentative images.

[0058] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0059] The present invention is applicable to the generation of spectral images. [Explanation of symbols]

[0060] 100...spectral image generating device, 10...encoded image acquisition unit, 20...control unit, 21...noise reduction unit, 211...temporary image reconstruction unit, 212...similar patch search unit, 213...group generation unit, 214...low-rank approximation unit, 215...patch integration unit, 216...iteration processing control unit, 22...image reconstruction unit, 30...encoded image storage unit, 80...encoded image, 81...similar encoded patch, 90...temporary image, 91...temporary patch, 92...similar provisional patch

Claims

1. a provisional image reconstruction unit that generates a provisional image by reconstructing an encoded image obtained by compressed spectral imaging; a similar patch search unit that acquires a plurality of similar temporary patches, which are small regions including images similar to each other, in the temporary image; a group generation unit that acquires a plurality of similar coded patches, which are small regions including images similar to each other in the coded image, based on information on positions where the similar temporary patches are acquired; a low-rank approximation unit that performs low-rank approximation based on a plurality of the similar coded patches to obtain a patch in which noise on the image is reduced in an area included in the similar coded patch; a patch integrating unit that integrates a plurality of noise-reduced patches according to information on the positions of the patches to generate an encoded image; A noise reduction device comprising:

2. The noise reduction device according to claim 1 , further comprising an iterative processing control unit that repeatedly executes at least the processing of said group generation unit and said low-rank approximation unit until a predetermined iteration termination condition is satisfied.

3. the tentative image reconstruction unit generates the tentative images for a plurality of coded images arranged in time series; The noise reduction device according to claim 1 , wherein the position information includes information on a spatial position within an image and information on a temporal position in the time series.

4. a provisional image reconstruction step of generating a provisional image by reconstructing an encoded image obtained by compressed spectral imaging; a similar patch search step of acquiring a plurality of similar temporary patches, which are small regions including images similar to each other, in the temporary image; a group generation step of acquiring a plurality of similar coded patches, which are small regions including images similar to each other in the coded image, based on information on positions where the similar temporary patches are acquired; a low-rank approximation step of performing low-rank approximation based on a plurality of the similar coded patches to obtain a patch in which noise on the image is reduced in an area included in the similar coded patch; a patch merging step of merging a plurality of noise-reduced patches according to position information of the patches to generate an encoded image; A noise reduction method comprising:

5. A computer program for causing a computer to function as the noise reduction device according to any one of claims 1 to 3.

Images